Mining rig extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS]

[rjk]

Good idea, I wasn't thinking. I'll need to see if the place has yellow (and orange? How much 3.3v do I actually need?) in the same type of wire available.

You could probably get by with smaller gauge wire on the 5 & 3.3 v lines too

That is correct sir!

I like how the wire is stiff enough to keep its shape for the long term. THHN uses large strands, and isn't all floppy like fine-stranded wire of lower gauge. Plus, it looks cool. I'm just putting my panel building skills to work in a different application 

Besides, I might be running much longer lengths than usual, so it helps with voltage drop too.

[1713524155]

1713524155

[1713524155]

1713524155

[1713524155]

1713524155

[1713524155]

1713524155

[rjk]

HASH 9000

Yes. This is perfect.

[check_status]

What socket type are the Xeon's?

[rjk]

What socket type are the Xeon's?

They are LV xeons - 2.8 Ghz, with package type being FC-mPGA4 (604-pin). So, socket 604.

[check_status]

Cool. I also found this.

Xeon LV 2.8
SL8RW (G0)
2800 MHz
1024 KiB
800 MT/s
14x
1.1125-1.2 V
55 W
Socket 604
October 2004
RK80546KG0721M

There are no single core socket 604 cpu's that support VT-x.  
VT-x isn't available until "Dempsey" Xeon 5020, 5030 for socket 771.

Edit: There are some dual core 604 sockets, Xeon 7020 that support VT-x. 165w
http://www.compuvest.com/Desc.jsp?iid=1124397  $30

[rjk]

There are no single core socket 604 cpu's that support VT-x.  
VT-x isn't available until "Dempsey" Xeon 5020, 5030 for socket 771.

Edit: There are some dual core 604 sockets, Xeon 7020 that support VT-x. 165w
http://www.compuvest.com/Desc.jsp?iid=1124397  $30

VT-x is hardware virtualization. VT-d is virtualization with directed I/O. Directed I/O means that a virtual machine can get full and unrestricted access to certain bits of hardware, such as video cards. That is what is needed to overcome the GPU limit in the drivers.

[check_status]

There can be some confusion regarding VT-d support on specific processors. VT-d is not a feature of the processor itself but rather a feature of the Chipset. As long as the chipset/BIOS combination support VT-d and the processor support VT, there will be VT-d support.

http://www.intel.com/support/processors/xeon5k/sb/CS-031637.htm

[rjk]

VT-d is not a feature of the processor itself but rather a feature of the Chipset. As long as the chipset/BIOS combination support VT-d and the processor support VT, there will be VT-d support.

http://www.intel.com/support/processors/xeon5k/sb/CS-031637.htm

I don't know how that could be correct. For instance, I know for a fact that the system I am typing on supports VT (i.e., VT-x), but not VT-d. It is a Z68 system. However, there is a chipset that does support VT-d, and it is the Q67 chipset. You can compare them here: http://ark.intel.com/compare/52812,52816

Compare these 2 processors, as well: http://ark.intel.com/compare/52213,52214
The 2600K certainly supports virtualization, but not VT-d. Here is a table to demonstrate this:

	P67	Z68
2600	VT-d	No
2600K	No	No

Perhaps that bit of info is specific to Xeon 5000 series or something? I don't know.

[check_status]

P67 and Z68 are the southbridge chipsets. My southbridge is SB850, which can run IOMMU but only with a BIOS from Gigabyte tech support.

I've learned a lot reading up on VT-d VT-x issue. Since the SBC uses a board manufactured pre-2005, the chipset on it doesn't support VT-d. According to Intel it requires the chipset to support IOMMU and the CPU to support VT functions.

I'll search around and see if I can find some PICMG with IOMMU.

[rjk]

P67 and Z68 are the southbridge chipsets. My southbridge is SB850, which can run IOMMU but only with a BIOS from Gigabyte tech support.

I've learned a lot reading up on VT-d VT-x issue. Since the SBC uses a board manufactured pre-2005, the chipset on it doesn't support VT-d. According to Intel it requires the chipset to support IOMMU and the CPU to support VT functions.

I'll search around and see if I can find some PICMG with IOMMU.

Cool, let me know if you find something for less than what I posted earlier. (An E3-1225 SHB for about $1600)

[DeathAndTaxes]

P67 and Z68 are the southbridge chipsets. My southbridge is SB850, which can run IOMMU but only with a BIOS from Gigabyte tech support.

I've learned a lot reading up on VT-d VT-x issue. Since the SBC uses a board manufactured pre-2005, the chipset on it doesn't support VT-d. According to Intel it requires the chipset to support IOMMU and the CPU to support VT functions.

I'll search around and see if I can find some PICMG with IOMMU.

Cool, let me know if you find something for less than what I posted earlier. (An E3-1225 SHB for about $1600)

What about something like this:
http://www.l-trondirect.com/pce-5124vg-00a1e.html?gclid=COWr5_GQuK4CFQ40hwodDC_-JA

$491.  Has Q35 chipset which supports VT-d.

[rjk]

P67 and Z68 are the southbridge chipsets. My southbridge is SB850, which can run IOMMU but only with a BIOS from Gigabyte tech support.

I've learned a lot reading up on VT-d VT-x issue. Since the SBC uses a board manufactured pre-2005, the chipset on it doesn't support VT-d. According to Intel it requires the chipset to support IOMMU and the CPU to support VT functions.

I'll search around and see if I can find some PICMG with IOMMU.

Cool, let me know if you find something for less than what I posted earlier. (An E3-1225 SHB for about $1600)

What about something like this:
http://www.l-trondirect.com/pce-5124vg-00a1e.html?gclid=COWr5_GQuK4CFQ40hwodDC_-JA

$491.  Has Q35 chipset which supports VT-d.

Hmmmmmmmm. I never considered that there could be other companies making the same things that would be compatible. I guess it is an open specification after all. (PICMG 1.3). I'll look into that and any similar alternatives, and see what I can budget for. Even though Q35 is a little old, it is way newer than what I have now.

EDIT: I just realized that I have been searching for PCIMG the whole time, not PICMG. 

[rjk]

So I've been looking around, but it appears that none of the available options come with processors. Instead of resurrecting the Core 2 series from the dead, it would probably be best to spend a hundred bucks more and upgrade to socket 1155. Bearing that in mind, how does this look for a parts list?

http://www.l-trondirect.com/pce-5126wg2-00a1e.html <---- SHB with C206 chipset $570.38
http://www.newegg.com/Product/Product.aspx?Item=N82E16819115088 <---- Xeon E3-1225 $219.99

If I wanted to stick to socket 775 and the Core 2 series, this seems to be a decent Q35 based board: http://www.ebay.com/itm/230658116351 $399
Then I would need to find a proc to fit that.

[check_status]

Looks like the chipset options on intel for VT-d are Q35, Q45, Q67, and C20*. So far your board choice looks pretty good.
http://forums.bit-tech.net/showthread.php?s=66d3b34a50d14b446bdb2987a3f069a1&p=2965518#post2965518

But you keyboard commandos are a little quicker than I am.

I wonder how the mapping would go with 16x 5970's, 4 per OS, conflicts?

Your board:
Q67 for QG2 version; B65 for QVG version; C206 for WG2 version

Some additional chipsets that have VT-d:
VT-d is enabled on the following chipsets:

   Intel Q35 GMCH with ICH9 DO (Bearlake chipset)

The following chipsets have VT-d capability, but OEMs may not have enabled in systems based on these:
   Intel X38
   Intel X48
VT-d will be enabled on these future products:
    Intel Q45 (Eaglelake)
For Intel Desktop Boards, these have VT-d support enabled:
   Intel DQ35JO
   Intel DQ35MP
   Intel DX38BT
   Intel DX48BT2
These future Intel Desktop Boards will have VT-d support:
   Intel DQ45CB
   Intel DQ45EK

[xurious]

http://www.newegg.com/Product/Product.aspx?Item=N82E16819115088 <---- Xeon E3-1225 $219.99

Splurge the 20 bucks at get the 1230 which supports hyperthreading. You're spending bucketloads on this already.

[rjk]

I wonder how the mapping would go with 16x 5970's, 4 per OS, conflicts?

Good question, I have no idea. From what I read so far, it may not even be able to boot unless I get a custom BIOS the doesn't initialize the cards on boot.

Splurge the 20 bucks at get the 1230 which supports hyperthreading. You're spending bucketloads on this already.

Totally, if it's only 20 bucks more. Cheap already at $220

[check_status]

I think I might have an interesting lead.
PICMG 1.3 board with a chipset that supports IOMMU and also the chipset can be built using Coreboot linux.
The board:
http://www.ryaim.com/products/singlebd/new_SBC/SPCIE-3600AM2.htm

This board uses the Nvidia MPC55 chipset, which Coreboot has been built in to, but for a Tyan S2912.
http://www.coreboot.org/Supported_Motherboards#Servers

If Coreboot is better to work with like for allocating resources for pci devices above 4G to expand the # of GPU's that are recognized similar to the fastra II project, that would be sweet.

[DiabloD3]

I'm not going to scroll through this thread to ask this question

Those two 8 pin plugs on the board, are those EPS12s or PCI-E 8s? Because if they're EPS12s (384w * 2) and they're wired to the PCI-E 12v rails, you have enough power for 5 5970s/6990s (ten slots worth, so you're still short four cards), or 18 cards of any kind (typically 25-35 watts off the slot, you have a budget of 42 watts).

If those are PCI-E 8s, then thats 150w each so.... 4 5970/6990s, or 9 or 10 normal cards.

I'm also assuming the computer is a low wattage model and only needs the 12v off the ATX24.

[rjk]

I'm not going to scroll through this thread to ask this question

Those two 8 pin plugs on the board, are those EPS12s or PCI-E 8s? Because if they're EPS12s (384w * 2) and they're wired to the PCI-E 12v rails, you have enough power for 5 5970s/6990s (ten slots worth, so you're still short four cards), or 18 cards of any kind (typically 25-35 watts off the slot, you have a budget of 42 watts).

If those are PCI-E 8s, then thats 150w each so.... 4 5970/6990s, or 9 or 10 normal cards.

I'm also assuming the computer is a low wattage model and only needs the 12v off the ATX24.

Those are 2x EPS12s. However, they get in the way of 2 or 3 slots when in use, which is why I am using the terminal blocks. The SHB I currently have uses 2x low voltage xeons, so not much power at all. However, additional power is needed by other parts on the board such as the PCIe fanout switches, and other SHBs can use hugely powerful CPUs such as dual hex core Xeons. So the board is rather overdesigned for power. (200 amps max at 12 VDC alone, just for the board).

[DiabloD3]

I'm not going to scroll through this thread to ask this question

Those two 8 pin plugs on the board, are those EPS12s or PCI-E 8s? Because if they're EPS12s (384w * 2) and they're wired to the PCI-E 12v rails, you have enough power for 5 5970s/6990s (ten slots worth, so you're still short four cards), or 18 cards of any kind (typically 25-35 watts off the slot, you have a budget of 42 watts).

If those are PCI-E 8s, then thats 150w each so.... 4 5970/6990s, or 9 or 10 normal cards.

I'm also assuming the computer is a low wattage model and only needs the 12v off the ATX24.

Those are 2x EPS12s. However, they get in the way of 2 or 3 slots when in use, which is why I am using the terminal blocks. The SHB I currently have uses 2x low voltage xeons, so not much power at all. However, additional power is needed by other parts on the board such as the PCIe fanout switches, and other SHBs can use hugely powerful CPUs such as dual hex core Xeons. So the board is rather overdesigned for power. (200 amps max at 12 VDC alone, just for the board).

Yeah, the board is basically perfect for mass mining... also, it doesn't really matter if two or three of the slots are blocked, you can only use 16 slots (ie, 8 cards) without fucking with virtualization, the last two are wasted anyways.