Bitcoin Forum
December 08, 2016, 02:32:39 AM *
News: To be able to use the next phase of the beta forum software, please ensure that your email address is correct/functional.
 
   Home   Help Search Donate Login Register  
Pages: [1] 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 »
  Print  
Author Topic: Mining rig extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS]  (Read 155741 times)
rjk
Sr. Member
****
Offline Offline

Activity: 420


1ngldh


View Profile
February 18, 2012, 08:49:37 PM
 #1

NOTE: Some info in this post is out of date. Read the thread for the most up-to-date info. I may create a new thread soon that has the info in a more concise format.

NOTE: All the parts and accessories for this build have been sold.

So I have been talking about this in other unrelated threads around here, and I figured it would probably be better to start my own topic for this project, to keep all the information in one place. There are obviously a large number of hurdles in this project that must be overcome before it will be useful, not the least of which is the driver limit of 8 GPUs. I will attempt to address these as I meander through the project - this is definitely a weekender since I am rather busy with daily work, so don't expect magical things to happen right away. Pics of the project will be kept in an Imgur album located here: http://imgur.com/a/0oist EDIT: Parts list and cost table will be kept updated in the second post in this thread!

This is my quest to get the insane and crazy 18 slot PCI Express backplane from Trenton Technology working with video cards for bitcoin mining. I was able to pick up the board for $600 - Trenton quoted me almost double that in single quantities. The one I got is unused.

To go with it, I need a Single Board Computer (SBC)/System Host Board (SHB) - Trenton prefers the notation SHB, so I will stick to that when referencing it. Basically, the board with all the slots is just an expansion board - the actual computing power gets plugged into it just the same as all the video cards (or whatever someone wishes to plug in). Since I do not need a lot of power to run miners, and I was trying to be cheap, I went with a refurbished NLT6313 (also from Trenton) which I obtained for $200. When it arrived, it appeared to be brand new, I couldn't tell that it had ever been used and then refurbished. To go with it, I got 8 GB of Hynix ECC DDR2 (4x 2GB) for $30. This board has two LV (Low Voltage) Xeon server processors, which should also help save some power.

For power, I got myself a Dell 2360 watt power supply from their current-generation blade servers - it cost me $95. I still need to figure out how to turn it on. Wink I will add more as necessary - this model is just a hair over 90% efficient at 50% load. [PDF link]

Here are some pics to kick off the thread - click them for full-size images:


This is the SHB and part of the backplane.


This shows where the SHB plugs into the backplane.


Full frontal including the SHB plugged into the backplane.


Another view.


Mission control, we have a problem! No way in hell are we going to fit a card into that last slot there....


IT MINES! CD drive and HDD disconnected a few minutes later, running on BAMT with 1 5850 for now.
Did you notice that it wasn't in the bottom slot? Yeah...


The PSU connector gets in the way of the bottom slot. Have we been reduced to only 16 useable slots? Not so fast....


It also obscures one of the ATX 12v connectors, and that is only a 5850. Imagine what a 5870 or 5970 would do!

Why am I not worried about the missing slots? Simple, the PSU connectors currently in use for testing are only the poor-man's connectors. See all the black terminal blocks across the back of the board in the third picture? That is where I will be putting all my power in, and those blocks are the same height as the PCIe Connectors. (Read: everything is going to fit just fine) Oh, and what about those pesky CPU fansinks getting in the way of the last slot? Well, I have a couple choices there: I could use that slot for a smaller card, or for a network/RAID/whatever card, or I could do away with the fansinks and replace them with something smaller. (Waterblocks, anyone?)

Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1481164359
Hero Member
*
Offline Offline

Posts: 1481164359

View Profile Personal Message (Offline)

Ignore
1481164359
Reply with quote  #2

1481164359
Report to moderator
rjk
Sr. Member
****
Offline Offline

Activity: 420


1ngldh


View Profile
February 18, 2012, 08:50:41 PM
 #2

I will use this second post to keep a running cost table of my expenditures on this system.

Initial parts purchase:
Code:
  Trenton BPX6806 backplane                          +   $600.00
   Trenton NLT6313 host board                         +   $210.00
   Hynix ECC DDR2 PC2-3200 4x 2GB RAM                 +    $33.95
   Dell 2360 watt blade server PSU                    +    $95.00

Total:                                                =   $938.95
This edit 2/19/2012 8:48 AM Eastern Updated 2/22/2012 9:10 PM Eastern with more accurate prices

-------------

More parts:
Code:
   50 FT 14 AWG THHN stranded wire - Red             +    $12.75
    50 FT 14 AWG THHN stranded wire - Black           +    $12.75
    75 Qty. Spade connectors                          +     $6.78

Subtotal:                                             =    $32.28

Previous total:                                       +   $938.95

Total:                                                =   $957.28
This edit 2/22/2012 9:10 PM Eastern

-------------

Zip ties and PSU connectors:
Code:
  100 Qty. Zip ties                                  +     $4.26
     8 Qty. Molex EXTreme PowerMass conn.             +    $40.78

Subtotal:                                             =    $45.04

Previous total:                                       +   $938.95

Total:                                                =   $983.99
This edit 2/23/2012 10:21 PM Eastern

-------------

More wire:
Code:
   50 FT 18-2 AWG solid wire - Red and White         +     $9.00
   100 FT 14 AWG THHN stranded wire - Yellow          +    $27.07
    50 FT 14 AWG THHN stranded wire - Orange          +    $25.94
   Molex connectors and pins, 20 pos barrier strip    +    $17.31

Subtotal:                                             =    $79.32

Previous total:                                       +   $983.99

Total:                                                = $1,063.31
This edit 3/3/2012 1:40 PM Eastern

-------------

Fans:
Code:
  8 Qty. Delta PFC1212DE-PWM 120x38mm Fan            +   $185.55

Previous total:                                       + $1,063.31

Total:                                                = $1,248.86
This edit 3/27/2012 10:30 Eastern

-------------

Lots more stuff that I have forgotten to add until just now:
Code:
    5 FT 6 AWG stranded battery cable - Yellow       +     $5.20
     5 FT 6 AWG stranded battery cable - Black        +     $6.20
     6 Qty. APC AP8716R locking power cord            +    $52.24
    16 Qty. 120mm Fan Grilles                         +    $19.29
    60 Qty. Fan Screws                                +    $12.09
   IBM DPI C19 PDU+ (40K9629)                         +    $74.84
   Arctic Silver ACN-60ML TIM remover & purifier      +     $7.99
   Tuniq TX-4 thermal grease                          +    $20.26
   Advantech PCE-5126WG2-00A1E                        + $1,109.89
      Includes Xeon E3-1225 and 8GB DDR3 ECC RAM
   Custom aluminum frame                              +   $309.00

Subtotal:                                             + $1,617.00

Previous total:                                       + $1,248.86

Total:                                                = $2,865.86
This edit 4/28/2012 3:17 Eastern

-------------

Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!
Unacceptable
Legendary
*
Offline Offline

Activity: 1946



View Profile
February 18, 2012, 10:24:14 PM
 #3

I had looked into this myself awhile back,but thought the power draw through the PCI Express slots with that many vid cards would melt the mobo Embarrassed

Hope you get it working,good luck!!!!!!!!!!

"If you run into an asshole in the morning, you ran into an asshole. If you run into assholes all day long, you are the asshole."  -Raylan Givens
Got GOXXED ?? https://www.youtube.com/watch?v=9KiqRpPiJAU&feature=youtu.be
"An ASIC being late is perfectly normal, predictable, and legal..."Hashfast & BFL slogan Smiley
rjk
Sr. Member
****
Offline Offline

Activity: 420


1ngldh


View Profile
February 18, 2012, 10:30:07 PM
 #4

I had looked into this myself awhile back,but thought the power draw through the PCI Express slots with that many vid cards would melt the mobo Embarrassed

Hope you get it working,good luck!!!!!!!!!!
According to Trenton's technical specifications, this board was specifically designed to support 75 watts per slot as per PCIe spec. The -004 version (which I have) has up to 200 amp input at 12 VDC available among all the power connectors. The board itself is extremely thick and durable - probably double the thickness of most motherboards.

Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!
bitlane
Internet detective
Sr. Member
****
Offline Offline

Activity: 462


I heart thebaron


View Profile
February 18, 2012, 10:41:26 PM
 #5

Are you being overly dramatic about slot locations and useability for the sake of being dramatic ? .....LOL

Any sane person would integrate this into a custom mining stand using 60-100% PCIe soft risers/cables, no ?

Every 3rd or so card on the board, then risers to attach others above to keep everything spaced and cool ? .....or am I to assume this is going into a rackmount case eventually ?

I must admit....this is awesome and I hope you get it working.

rjk
Sr. Member
****
Offline Offline

Activity: 420


1ngldh


View Profile
February 18, 2012, 10:54:23 PM
 #6

Are you being overly dramatic about slot locations and useability for the sake of being dramatic ? .....LOL

Any sane person would integrate this into a custom mining stand using 60-100% PCIe soft risers/cables, no ?

Every 3rd or so card on the board, then risers to attach others above to keep everything spaced and cool ? .....or am I to assume this is going into a rackmount case eventually ?

I must admit....this is awesome and I hope you get it working.
The idea was for it to be rackmount, otherwise you would be able to build the same mining power into a larger space easily. Its all about density here. Wink

The other thing that I mentioned in another thread is that it may be a better idea to build it with an enterprise in mind - not Bitcoins. To that end, it would make more sense to use something like nVidia Quadro cards, or AMD FirePro stuff. For mining though.... 5970s all the way.

Another small hang-up: In researching waterblocks that fit 5870s and 5970s, they all seem to be 0.7 to 0.9 inches thick. Since PCIe slot spacing is 0.8 inches, that wouldn't fit. Does anyone know of waterblocks that are known to fit side-by-side in adjacent slots? Or will I have to get them made custom.... Danger Den, I'm looking at you...

Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!
The-Real-Link
Hero Member
*****
Offline Offline

Activity: 530



View Profile
February 18, 2012, 11:15:36 PM
 #7

That's an epic board!  Good luck fitting it out and continuing to make it all work Wink.  Very impressive.

Oh Loaded, who art up in Mt. Gox, hallowed be thy name!  Thy dollars rain, thy will be done, on BTCUSD.  Give us this day our daily 10% 30%, and forgive the bears, as we have bought their bitcoins.  And lead us into quadruple digits
bulanula
Hero Member
*****
Offline Offline

Activity: 518



View Profile
February 18, 2012, 11:53:26 PM
 #8

Well done rjk for investigating this crazy land of expensive backplanes.

We bow down to your awesomeness Shocked

Keep up the good work and us posted !
ZodiacDragon84
Sr. Member
****
Offline Offline

Activity: 266


The king and the pawn go in the same box @ endgame


View Profile
February 19, 2012, 01:07:16 AM
 #9

i must watch rjk's shenanigans, and learn

Looking for a quick easy mining solution? Check out
www.bitminter.com

See my trader rep at Bitcoinfeedback.com
!
sveetsnelda
Hero Member
*****
Offline Offline

Activity: 644


View Profile
February 19, 2012, 01:11:57 AM
 #10

Definitely watching this.  Smiley

14u2rp4AqFtN5jkwK944nn741FnfF714m7
eviltt
Member
**
Offline Offline

Activity: 84


View Profile
February 19, 2012, 06:28:02 AM
 #11

RJK,

Are you just running BAMT native on that Machine or are you virtualizing it as I assume you will have to in order to get beyond the 8GPU limit per OS?
check_status
Full Member
***
Offline Offline

Activity: 196


Web Dev, Db Admin, Computer Technician


View Profile
February 19, 2012, 07:59:56 AM
 #12

Yep, rjk's Battle of Trenton will be even more epic than the Revolutionary War with General George Washington.

A George Washington smile for rjk.


When the 8 GPU barrier is broken by the diligent open source peeps rjk will be ahead of the game.

For Bitcoin to be a true global currency the value of BTC needs always to rise.
If BTC became the global currency & money supply = 100 Trillion then ⊅1.00 BTC = $4,761,904.76.
P2Pool Server List | How To's and Guides Mega List |  1EndfedSryGUZK9sPrdvxHntYzv2EBexGA
rjk
Sr. Member
****
Offline Offline

Activity: 420


1ngldh


View Profile
February 19, 2012, 01:34:30 PM
 #13

RJK,

Are you just running BAMT native on that Machine or are you virtualizing it as I assume you will have to in order to get beyond the 8GPU limit per OS?


No right now I just got it going really quick yesterday, haven't spent much time on it yet. My plan was to virtualize it if I couldn't make something else work with the drivers. However after a very hasty peek at the BIOS, there doesn't seem to be options for enabling/disabling VT-x and VT-d. However I might have missed something obvious in my haste.

Yep, rjk's Battle of Trenton will be even more epic than the Revolutionary War with General George Washington.

A George Washington smile for rjk.


When the 8 GPU barrier is broken by the diligent open source peeps rjk will be ahead of the game.
Lol!

I need to figure this virtualization situation out. Perhaps I can look up the processors in the Intel Ark. They are 2.8 Ghz LV (Low Voltage) Xeons, but I'm not sure of a model number.

I have also heard someone say that the display drivers in linux are loaded for each X session, and that I could run multiple X sessions? This seems far-fetched though.... anyone tried this before?

Finally, this system has a very intriguing BIOS option: Namely, the ability to switch between 32-bit and 64-bit PCIe resource tables. I think this is to overcome some kind of addressing limitations that apply to having so many slots, but enabling 64-bit mode made it so that neither BAMT nor Windows would load/install. Both reduced the display down to 4 colors (or was it 16?), and Windows wouldn't install without a driver. It didn't tell me what device the driver it needed was for though, so anything I presented to it was rejected.

Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!
dizzy1
Full Member
***
Offline Offline

Activity: 134


View Profile
February 19, 2012, 08:14:55 PM
 #14

http://wiki.xen.org/wiki/XenVGAPassthrough
rjk
Sr. Member
****
Offline Offline

Activity: 420


1ngldh


View Profile
February 19, 2012, 09:18:52 PM
 #15

I wish. I did some research, and as far as I can determine, I have these processors: http://ark.intel.com/products/33103/64-bit-Intel-Xeon-Processor-2_80-GHz-2M-Cache-800-MHz-FSB

Yeah they support 64-bit processing and hyperthreading, but that is about the only modern (ish?) features that they have.. Definitely no VT-d (needed for Xen VGA Passthrough), and not even VT-x (needed for any kind of virtualization that runs on bare metal).

The cheapest "new" option that supports VT-d is the Intel Xeon E3-1225 - which Trenton quoted me at $1,610 in single quantities.  Angry If any of you have access to TSB or JXT series SHB cards, or know where to get them at a reasonable price, do please let me know...

Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!
The-Real-Link
Hero Member
*****
Offline Offline

Activity: 530



View Profile
February 19, 2012, 10:14:14 PM
 #16

Seems like you could do 5 5970s in that with 7 single slot cards. 

Oh Loaded, who art up in Mt. Gox, hallowed be thy name!  Thy dollars rain, thy will be done, on BTCUSD.  Give us this day our daily 10% 30%, and forgive the bears, as we have bought their bitcoins.  And lead us into quadruple digits
ssateneth
Legendary
*
Offline Offline

Activity: 1288



View Profile
February 19, 2012, 10:42:56 PM
 #17

Seems like you could do 5 5970s in that with 7 single slot cards. 

Might want to read the thread. We still have to deal with 8 gpu driver limit.

bulanula
Hero Member
*****
Offline Offline

Activity: 518



View Profile
February 19, 2012, 10:44:09 PM
 #18

Seems like you could do 5 5970s in that with 7 single slot cards. 

Might want to read the thread. We still have to deal with 8 gpu driver limit.

Yeah might want to pressure ATI to remove this stupid ARTIFICIAL limit in their crap drivers.

If more of us do it, then they will probably take their heads out of their asses and listen.
gmaxwell
Moderator
Legendary
*
Offline Offline

Activity: 2030



View Profile
February 19, 2012, 11:10:17 PM
 #19

Seems like you could do 5 5970s in that with 7 single slot cards. 
Might want to read the thread. We still have to deal with 8 gpu driver limit.

This can be worked around quite reasonably.

rjk
Sr. Member
****
Offline Offline

Activity: 420


1ngldh


View Profile
February 19, 2012, 11:15:55 PM
 #20

Seems like you could do 5 5970s in that with 7 single slot cards.  
Might want to read the thread. We still have to deal with 8 gpu driver limit.

This can be worked around quite reasonably.


Do tell? If it is anything with running multiple X sessions, I am enough of a linux noob to not know how this works or where to begin. If you mean by using VT-d and some virtual machines to overcome the driver limits, sure sounds good but my hardware is not capable (no VT-x or VT-d unless I get a more expensive host board). Or, if you mean modifying the drivers in any way, or using specific versions without a limit, perhaps you will enlighten me in that regard?

EDIT: Spoke on IRC, and it seems that VT-d was the idea, but unfortunately my current hardware doesn't support it Sad

Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!
Pages: [1] 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 »
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!