Mining rig extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS]

[rjk]

NOTE: Some info in this post is out of date. Read the thread for the most up-to-date info. I may create a new thread soon that has the info in a more concise format.

NOTE: All the parts and accessories for this build have been sold.

So I have been talking about this in other unrelated threads around here, and I figured it would probably be better to start my own topic for this project, to keep all the information in one place. There are obviously a large number of hurdles in this project that must be overcome before it will be useful, not the least of which is the driver limit of 8 GPUs. I will attempt to address these as I meander through the project - this is definitely a weekender since I am rather busy with daily work, so don't expect magical things to happen right away. Pics of the project will be kept in an Imgur album located here: http://imgur.com/a/0oist EDIT: Parts list and cost table will be kept updated in the second post in this thread!

This is my quest to get the insane and crazy 18 slot PCI Express backplane from Trenton Technology working with video cards for bitcoin mining. I was able to pick up the board for $600 - Trenton quoted me almost double that in single quantities. The one I got is unused.

To go with it, I need a Single Board Computer (SBC)/System Host Board (SHB) - Trenton prefers the notation SHB, so I will stick to that when referencing it. Basically, the board with all the slots is just an expansion board - the actual computing power gets plugged into it just the same as all the video cards (or whatever someone wishes to plug in). Since I do not need a lot of power to run miners, and I was trying to be cheap, I went with a refurbished NLT6313 (also from Trenton) which I obtained for $200. When it arrived, it appeared to be brand new, I couldn't tell that it had ever been used and then refurbished. To go with it, I got 8 GB of Hynix ECC DDR2 (4x 2GB) for $30. This board has two LV (Low Voltage) Xeon server processors, which should also help save some power.

For power, I got myself a Dell 2360 watt power supply from their current-generation blade servers - it cost me $95. I still need to figure out how to turn it on. I will add more as necessary - this model is just a hair over 90% efficient at 50% load. [PDF link]

Here are some pics to kick off the thread - click them for full-size images:


This is the SHB and part of the backplane.


This shows where the SHB plugs into the backplane.


Full frontal including the SHB plugged into the backplane.


Another view.


Mission control, we have a problem! No way in hell are we going to fit a card into that last slot there....


IT MINES! CD drive and HDD disconnected a few minutes later, running on BAMT with 1 5850 for now.
Did you notice that it wasn't in the bottom slot? Yeah...


The PSU connector gets in the way of the bottom slot. Have we been reduced to only 16 useable slots? Not so fast....


It also obscures one of the ATX 12v connectors, and that is only a 5850. Imagine what a 5870 or 5970 would do!

Why am I not worried about the missing slots? Simple, the PSU connectors currently in use for testing are only the poor-man's connectors. See all the black terminal blocks across the back of the board in the third picture? That is where I will be putting all my power in, and those blocks are the same height as the PCIe Connectors. (Read: everything is going to fit just fine) Oh, and what about those pesky CPU fansinks getting in the way of the last slot? Well, I have a couple choices there: I could use that slot for a smaller card, or for a network/RAID/whatever card, or I could do away with the fansinks and replace them with something smaller. (Waterblocks, anyone?)

[rjk]

I will use this second post to keep a running cost table of my expenditures on this system.

Initial parts purchase:

Code:

   Trenton BPX6806 backplane                          +   $600.00
   Trenton NLT6313 host board                         +   $210.00
   Hynix ECC DDR2 PC2-3200 4x 2GB RAM                 +    $33.95
   Dell 2360 watt blade server PSU                    +    $95.00

Total:                                                =   $938.95

This edit 2/19/2012 8:48 AM Eastern Updated 2/22/2012 9:10 PM Eastern with more accurate prices

-------------

More parts:

Code:

    50 FT 14 AWG THHN stranded wire - Red             +    $12.75
    50 FT 14 AWG THHN stranded wire - Black           +    $12.75
    75 Qty. Spade connectors                          +     $6.78

Subtotal:                                             =    $32.28

Previous total:                                       +   $938.95

Total:                                                =   $957.28

This edit 2/22/2012 9:10 PM Eastern

-------------

Zip ties and PSU connectors:

Code:

   100 Qty. Zip ties                                  +     $4.26
     8 Qty. Molex EXTreme PowerMass conn.             +    $40.78

Subtotal:                                             =    $45.04

Previous total:                                       +   $938.95

Total:                                                =   $983.99

This edit 2/23/2012 10:21 PM Eastern

-------------

More wire:

Code:

    50 FT 18-2 AWG solid wire - Red and White         +     $9.00
   100 FT 14 AWG THHN stranded wire - Yellow          +    $27.07
    50 FT 14 AWG THHN stranded wire - Orange          +    $25.94
   Molex connectors and pins, 20 pos barrier strip    +    $17.31

Subtotal:                                             =    $79.32

Previous total:                                       +   $983.99

Total:                                                = $1,063.31

This edit 3/3/2012 1:40 PM Eastern

-------------

Fans:

Code:

   8 Qty. Delta PFC1212DE-PWM 120x38mm Fan            +   $185.55

Previous total:                                       + $1,063.31

Total:                                                = $1,248.86

This edit 3/27/2012 10:30 Eastern

-------------

Lots more stuff that I have forgotten to add until just now:

Code:

     5 FT 6 AWG stranded battery cable - Yellow       +     $5.20
     5 FT 6 AWG stranded battery cable - Black        +     $6.20
     6 Qty. APC AP8716R locking power cord            +    $52.24
    16 Qty. 120mm Fan Grilles                         +    $19.29
    60 Qty. Fan Screws                                +    $12.09
   IBM DPI C19 PDU+ (40K9629)                         +    $74.84
   Arctic Silver ACN-60ML TIM remover & purifier      +     $7.99
   Tuniq TX-4 thermal grease                          +    $20.26
   Advantech PCE-5126WG2-00A1E                        + $1,109.89
      Includes Xeon E3-1225 and 8GB DDR3 ECC RAM
   Custom aluminum frame                              +   $309.00

Subtotal:                                             + $1,617.00

Previous total:                                       + $1,248.86

Total:                                                = $2,865.86

This edit 4/28/2012 3:17 Eastern

-------------

[Unacceptable]

I had looked into this myself awhile back,but thought the power draw through the PCI Express slots with that many vid cards would melt the mobo

Hope you get it working,good luck!!!!!!!!!!

[rjk]

I had looked into this myself awhile back,but thought the power draw through the PCI Express slots with that many vid cards would melt the mobo

Hope you get it working,good luck!!!!!!!!!!

According to Trenton's technical specifications, this board was specifically designed to support 75 watts per slot as per PCIe spec. The -004 version (which I have) has up to 200 amp input at 12 VDC available among all the power connectors. The board itself is extremely thick and durable - probably double the thickness of most motherboards.

[bitlane]

Are you being overly dramatic about slot locations and useability for the sake of being dramatic ? .....LOL

Any sane person would integrate this into a custom mining stand using 60-100% PCIe soft risers/cables, no ?

Every 3rd or so card on the board, then risers to attach others above to keep everything spaced and cool ? .....or am I to assume this is going into a rackmount case eventually ?

I must admit....this is awesome and I hope you get it working.

[rjk]

Are you being overly dramatic about slot locations and useability for the sake of being dramatic ? .....LOL

Any sane person would integrate this into a custom mining stand using 60-100% PCIe soft risers/cables, no ?

Every 3rd or so card on the board, then risers to attach others above to keep everything spaced and cool ? .....or am I to assume this is going into a rackmount case eventually ?

I must admit....this is awesome and I hope you get it working.

The idea was for it to be rackmount, otherwise you would be able to build the same mining power into a larger space easily. Its all about density here.

The other thing that I mentioned in another thread is that it may be a better idea to build it with an enterprise in mind - not Bitcoins. To that end, it would make more sense to use something like nVidia Quadro cards, or AMD FirePro stuff. For mining though.... 5970s all the way.

Another small hang-up: In researching waterblocks that fit 5870s and 5970s, they all seem to be 0.7 to 0.9 inches thick. Since PCIe slot spacing is 0.8 inches, that wouldn't fit. Does anyone know of waterblocks that are known to fit side-by-side in adjacent slots? Or will I have to get them made custom.... Danger Den, I'm looking at you...

[The-Real-Link]

That's an epic board!  Good luck fitting it out and continuing to make it all work .  Very impressive.

[bulanula]

Well done rjk for investigating this crazy land of expensive backplanes.

We bow down to your awesomeness

Keep up the good work and us posted !

[ZodiacDragon84]

i must watch rjk's shenanigans, and learn

[sveetsnelda]

Definitely watching this. 

[eviltt]

RJK,

Are you just running BAMT native on that Machine or are you virtualizing it as I assume you will have to in order to get beyond the 8GPU limit per OS?

[check_status]

Yep, rjk's Battle of Trenton will be even more epic than the Revolutionary War with General George Washington.

A George Washington smile for rjk.


When the 8 GPU barrier is broken by the diligent open source peeps rjk will be ahead of the game.

[rjk]

RJK,

Are you just running BAMT native on that Machine or are you virtualizing it as I assume you will have to in order to get beyond the 8GPU limit per OS?

No right now I just got it going really quick yesterday, haven't spent much time on it yet. My plan was to virtualize it if I couldn't make something else work with the drivers. However after a very hasty peek at the BIOS, there doesn't seem to be options for enabling/disabling VT-x and VT-d. However I might have missed something obvious in my haste.

Yep, rjk's Battle of Trenton will be even more epic than the Revolutionary War with General George Washington.

A George Washington smile for rjk.


When the 8 GPU barrier is broken by the diligent open source peeps rjk will be ahead of the game.

Lol!

I need to figure this virtualization situation out. Perhaps I can look up the processors in the Intel Ark. They are 2.8 Ghz LV (Low Voltage) Xeons, but I'm not sure of a model number.

I have also heard someone say that the display drivers in linux are loaded for each X session, and that I could run multiple X sessions? This seems far-fetched though.... anyone tried this before?

Finally, this system has a very intriguing BIOS option: Namely, the ability to switch between 32-bit and 64-bit PCIe resource tables. I think this is to overcome some kind of addressing limitations that apply to having so many slots, but enabling 64-bit mode made it so that neither BAMT nor Windows would load/install. Both reduced the display down to 4 colors (or was it 16?), and Windows wouldn't install without a driver. It didn't tell me what device the driver it needed was for though, so anything I presented to it was rejected.

[dizzy1]

http://wiki.xen.org/wiki/XenVGAPassthrough

[rjk]

http://wiki.xen.org/wiki/XenVGAPassthrough

I wish. I did some research, and as far as I can determine, I have these processors: http://ark.intel.com/products/33103/64-bit-Intel-Xeon-Processor-2_80-GHz-2M-Cache-800-MHz-FSB

Yeah they support 64-bit processing and hyperthreading, but that is about the only modern (ish?) features that they have.. Definitely no VT-d (needed for Xen VGA Passthrough), and not even VT-x (needed for any kind of virtualization that runs on bare metal).

The cheapest "new" option that supports VT-d is the Intel Xeon E3-1225 - which Trenton quoted me at $1,610 in single quantities.   If any of you have access to TSB or JXT series SHB cards, or know where to get them at a reasonable price, do please let me know...

[The-Real-Link]

Seems like you could do 5 5970s in that with 7 single slot cards. 

[ssateneth]

Seems like you could do 5 5970s in that with 7 single slot cards. 

Might want to read the thread. We still have to deal with 8 gpu driver limit.

[bulanula]

Seems like you could do 5 5970s in that with 7 single slot cards. 

Might want to read the thread. We still have to deal with 8 gpu driver limit.

Yeah might want to pressure ATI to remove this stupid ARTIFICIAL limit in their crap drivers.

If more of us do it, then they will probably take their heads out of their asses and listen.

[gmaxwell]

Seems like you could do 5 5970s in that with 7 single slot cards. 

Might want to read the thread. We still have to deal with 8 gpu driver limit.

This can be worked around quite reasonably.

[rjk]

Seems like you could do 5 5970s in that with 7 single slot cards.  

Might want to read the thread. We still have to deal with 8 gpu driver limit.

This can be worked around quite reasonably.

Do tell? If it is anything with running multiple X sessions, I am enough of a linux noob to not know how this works or where to begin. If you mean by using VT-d and some virtual machines to overcome the driver limits, sure sounds good but my hardware is not capable (no VT-x or VT-d unless I get a more expensive host board). Or, if you mean modifying the drivers in any way, or using specific versions without a limit, perhaps you will enlighten me in that regard?

EDIT: Spoke on IRC, and it seems that VT-d was the idea, but unfortunately my current hardware doesn't support it