Mining rig extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS]

[rjk]

Have you enabled IOMMU in the BIOS?  It is my understanding that is necessary for PCIe pass-thru.

IOMMU is for AMD; I have an Intel platform so I have VT-d. And yes, it is enabled. PVE kernel panics when I try to hotplug the devices to the virtual machine, and the virtual machine won't boot when the devices are added to the config file when it isn't booted.

[1713477674]

1713477674

[1713477674]

1713477674

[rjk]

have you tried ESX(i)? i think it may be a little bit more user friendly, vCenter management software, just install it to an USB stick, and you should be able to create VMS (if you have some storage repository), after you have done that, you can (try to) add devices with the configuration manager (vcenter), you can do that from the start but i always check if it is working the normal way...   

why did you choose PVE? you feel more comfortable with it?

It was a recommendation, because it is based on a pre-setup KVM kernel. I heard ESXi has a limitation of 4 devices though?

[DiabloD3]

Have you enabled IOMMU in the BIOS?  It is my understanding that is necessary for PCIe pass-thru.

IOMMU is for AMD; I have an Intel platform so I have VT-d. And yes, it is enabled. PVE kernel panics when I try to hotplug the devices to the virtual machine, and the virtual machine won't boot when the devices are added to the config file when it isn't booted.

No. IOMMU is a industry term for it. VT-d is an implementation of IOMMU, AMD, Sun, and IBM also offer implementations of it.

[DiabloD3]

http://en.wikipedia.org/wiki/IOMMU#IOMMU_in_relation_to_virtualization   

AMD has published a specification for IOMMU technology in the HyperTransport architecture.[1] Intel has published a specification for IOMMU technology as Virtualization Technology for Directed I/O, abbreviated VT-d.[2] Information about the Sun IOMMU has been published in the Device Virtual Memory Access (DVMA) section of the Solaris Developer Connection.[3] The IBM Translation Control Entry (TCE) has been described in a document entitled Logical Partition Security in the IBM eServer pSeries 690.[4] The PCI-SIG has relevant work under the terms I/O Virtualization (IOV)[5] and Address Translation Services (ATS).

So, are we done playing wikipedia lawyer?

[rjk]

Anything new to report? If you get a kernel panic on a linux guest during boot or just can't get a linux guest to boot, I'd be interested in knowing if a similarly configured windows VM does the same.I think we had figured out about the device id's for your devices last.

It was the host that panicked, not the guest. The output is on the previous page.

Maybe ill try a windoz guest and see what happens.

[rjk]

I'm on my phone so it makes it a little hard to elaborate on all the points, but I'll try to remember all of them.

Yes there is onboard video, it doesn't show in that list because - grepped for "Cypress XT" not "VGA". When I grepped for VGA, I got several screens of output and didn't sift through it thoroughly.

I have tried hotplugging and coldplugging different devices. Yes, I would assume that the host is trying to use one of the cards and that could cause a conflict. I just had a thought - would it be possible that the host is glomming on all the cards just because they are there? I should change my BIOS to boot the internal video first and use it as primary display.

When hotplugging, the host (dom0) was what panicked. When coldplugging, the guest (domU) either flashed the VNC terminal on and off repeatedly (when a single device was configured) or showed a black screen and froze (no networking or anything) when all cards were assigned.

Unfortunately development is kind of on hold because of time constraints (weekender) and because these fans at 100% are damn annoying.

I need to get the following list of stuff done:
Custom bracket - need to find a small fabricator near me
Copper bus bar - determine correct size, layout, and method of attaching conductors
PSU - figure out how to turn it on
PCIe 6/8 pin connectors - obtain a bunch and figure out how to hook them to the bus bars (ring/spade terminals? solder? clamp?)
Air channel/duct - needed so that the air doesn't escape past the heatsinks over the top - also a job for the fabricator - needs to be slotted so the upper level PCIe extenders can pass thru
Fan contoroller - wire it in once it arrives and set up the control software (to be written, coder is lined up for the job)

Damn, I wish it was as easy as throwing some money at it and putting it together like Legos. Problem is, I'm kinda weak when it comes to the software and scripting/programming side of things. Once it's done, does anyone wanna buy it and have the distinction of owning the worlds fastest, most complicated GPU based rig? It will need a 7KW power outlet and cooling.

[rjk]

one quick note, after what you said i remember you have to unbind the pci device first (logical right), have you done that? i cant recall having read it(but that doesnt say too much)

4. unbind device from host kernel driver (example PCI device 01:00.0)
 Load the PCI Stub Driver if it is compiled as a module
       modprobe pci_stub
 lspci -n
locate the entry for device 01:00.0 and note down the vendor & device ID 8086:10b9
        ...
       01:00.0 0200: 8086:10b9 (rev 06)
       ...
 echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind

http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM


in xen it was something with pciback

http://wiki.xensource.com/xenwiki/Assign_hardware_to_DomU_with_PCIBack_as_module

really not sure though..

Having to unbind it from the host first makes perfect sense, I didn't really think of that. I'll have to try those steps and see how I get on.

[rjk]

Having to unbind it from the host first makes perfect sense, I didn't really think of that. I'll have to try those steps and see how I get on.

Played with it a little more.
Unbinding and rebinding to PCI_stub seemed to work (Is this persistent? If not, how do I make it save these settings between reboots?), and I was able to attach a device to a VM. However, X segfaulted with error 11. I was able to lspci and see the video card from inside the VM though. PVE doesn't seem to like having more than 2 PCI devices per VM, and if you add more to the VM config, it simply removes them for you.

I tried installing a new VM with the cards pre-attached, and the bootdisk just showed a blank screen and went no further. I tried installing Windows 7 x64, but it kept asking for a driver for the CDROM which I didn't have or know where to get.

[rjk]

What's the smallest Linux install that I can have that will mine at full speed, and how little RAM can I assign to it? If I have to have 2 devices per VM, that would mean 17 VMs in a dual-GPU situation, which would suck. And, since I have only 8GB of RAM, I will have to have fewer than 512MB RAM per VM. I don't know how much RAM PVE uses for Dom0, but if I assume 1GB (like XenServer), that leaves me with 7GB divided by 17 VMs, equals ~420MB RAM per VM. I suppose that should be enough for something that is stripped down, but I doubt it would run BAMT with all its monitoring and stuff.

[Gomeler]

LinuxCoin maybe? although it doesn't say how much RAM you need

http://www.linuxcoin.co.uk/wiki/index.php/Headless_Linuxcoin

The disk image contains the AMD APP SDK, ATI Catalyst & Drivers, poclbm & phoenix miner, Bitcoin, and more.

the Xeon E3-1225  seems to have just 4 cores & 4 threads .. and i think the host wants one , so maximum 3 virtual machines possible, each getting a thread?

You can overprovision the CPU on a VM host. So long as you don't peg the CPU cores on all your VMs, the hypervisor will load balance your CPUs to provide enough CPU power for your VMs. The major obstacle is always ram and storage IOPs. Luckily ram is dirt cheap.

[Gomeler]

If ram becomes an issue I'd suggest just trying 8 GB dimms. It may work, I've seen more bizarre things than that.

[terrytibbs]

luke-jr is virtualizing gpu's with kvm. I'll try to dig up that info or have a conversation with him or see how he likes this thread.

Just don't mention religion.

[rjk]

luke-jr is virtualizing gpu's with kvm. I'll try to dig up that info or have a conversation with him or see how he likes this thread.

You know, I did discuss it with him months ago, but that was only the most basic bare minimum of information, because I didn't have compatible hardware. I'll need to grep my IRC logs to see if he gave out any technical information. All I remember was that it was 1 GPU and that it kept crashing.

EDIT: I found some stuff, and I know there is more, but I can't be bothered to find it all right now.

Code:

Dec 26 19:54:12 <rjk2> gmaxwell: i thought there was a driver limit of 8 gpus maximum
Dec 26 19:54:28 <gmaxwell> Driver. Yup.
Dec 26 19:54:35 <gmaxwell> So run multiple copies of the driver. :)
Dec 26 19:54:55 <gmaxwell> (according to luke the ati drivers work in kvm with the pci remapping stuff)
Dec 26 19:55:35 <gmaxwell> so in _theory_ you could start two VMs, map 4 cards to each and run 16 GPUs on a system.
Dec 26 19:55:45 <gmaxwell> (presuming you could plug in that many)
Dec 26 19:55:58 <gmaxwell> (and presuming your motherboard doesn't catch fire)
Dec 26 19:57:38 <rjk2> I have been trying to find someone that has actually been able to mine successfully in a virtualized environment
Dec 26 19:57:48 <gmaxwell> luke-jr:
Dec 26 19:57:52 <rjk2> i know it ought to work
Dec 26 19:58:16 <luke-jr> gmaxwell: only with motherboards/CPUs that support it
Dec 26 19:58:44 <rjk2> vt-d, and iommu right
Dec 26 19:58:49 <luke-jr> VT-d in my case
Dec 26 19:58:54 <rjk2> but does it work well?
Dec 26 19:58:55 <rjk2> stable?
Dec 26 19:58:58 <luke-jr> also, Intel pulled a big scam earlier this year
Dec 26 19:59:02 <rjk2> because i want to try it
Dec 26 19:59:09 <luke-jr> well, my Radeon still manages to crash my host system sometimes
Dec 26 19:59:17 <rjk2> hmm
Dec 26 19:59:25 <gmaxwell> rjk2: you get diminishing returns packing more cards onto one system. ::shrugs::
Dec 26 19:59:26 <luke-jr> usually when I try to reinitialize it
Dec 26 19:59:30 <luke-jr> which is mostly playing around
Dec 26 19:59:38 <luke-jr> if I knew a proper way to do that, it might be OK
Dec 26 20:00:09 <rjk2> what i wanted to do was use intel vt-d with  built-in video powering my desktop environment, and discrete cards mining
Dec 26 20:00:23 <luke-jr> rjk2: tht's what I do
Dec 26 20:00:24 <rjk2> z68 based
Dec 26 20:00:46 <rjk2> luke-jr: sounds good, using kvm or xen?
Dec 26 20:01:21 <luke-jr> KVM
Dec 26 20:01:47 <luke-jr> Z68 = no VT-d
Dec 26 20:01:53 <rjk2> wait what
Dec 26 20:02:05 * catalase_ is now known as catalase
Dec 26 20:02:08 <rjk2> i know the K procs with overclocking don't have it
Dec 26 20:02:17 <rjk2> but z68 doesn't either?
Dec 26 20:02:20 <luke-jr> nope
Dec 26 20:02:26 <luke-jr> http://ark.intel.com/compare/52816,52812
Dec 26 20:02:31 <rjk2> damn it
Dec 26 20:02:37 <luke-jr> return it :P
Dec 26 20:03:13 <rjk2> is that what "embedded options available" means? VT-d?
Dec 26 20:03:19 <luke-jr> no
Dec 26 20:03:26 <luke-jr> Intel® Virtualization Technology for Directed I/O (VT-d)
Dec 26 20:03:37 <luke-jr> it's exclusive to Q67
Dec 26 20:03:38 <rjk2> oh, i see its down lower
Dec 26 20:03:41 <luke-jr> at least for Sandy Bridge
Dec 26 20:03:45 <rjk2> hmm
Dec 26 20:04:01 <luke-jr> originally it wasn't
Dec 26 20:04:03 <rjk2> man that is lame
Dec 26 20:04:08 <luke-jr> but Intel bugged, and decided the solution was to fix their specs
Dec 26 20:04:20 <luke-jr> ie, they said the bug was the advertising VT-d on the others
Dec 26 20:04:31 <gmaxwell> doh! that sucks.
Dec 26 20:04:34 <luke-jr> yep
Dec 26 20:04:38 <luke-jr> but they also bugged the SATA
Dec 26 20:04:46 <luke-jr> so I returned my H67 and got a Q67
Dec 26 20:04:48 <luke-jr> :D
Dec 26 20:04:53 <rjk2> lol too true >.>
Dec 26 20:04:58 <luke-jr> though it was VERY difficult to get Q67 at the time
Dec 26 20:07:34 <rjk2> luke-jr: have you had the opportunity to test IOMMU (AMD) systems in the same manner yet?
Dec 26 20:08:21 <luke-jr> no
Dec 26 20:08:36 <luke-jr> I think someone did though, and found they had to do 1 VM per card
Dec 26 20:08:42 <rjk2> lolwut
Dec 26 20:08:44 <luke-jr> I've only tested 1 on VT-d
Dec 26 20:09:18 <luke-jr> the annoying thing is, if the card crashes, I pretty much have to reboot
Dec 26 20:09:33 <luke-jr> in theory, I *should* be able to 'remove' the card and power cycle just it
Dec 26 20:09:39 <luke-jr> but I haven't figured out how in practice
Dec 26 20:09:51 <luke-jr> even if I remove and rescan, it doesn't get reset
Dec 26 20:10:09 <luke-jr> upgrading fglrx *might* help
Dec 26 20:10:28 <luke-jr> since newer versions support hot-switching between IGP and discrete on laptops
Dec 26 20:10:58 <luke-jr> (the error I was getting found only Google results talking about that)
Dec 26 20:11:37 <luke-jr> latest fglrx *seems* to be working just as well as the good old 2.1 now tho
Dec 26 20:12:19 <luke-jr> ie, no CPU hogging, and full 308 MH/s
Dec 26 20:12:48 <luke-jr> I don't have a sane way to reproduce the card crashing though, so no idea how many months until I can confirm it reinitializes
Dec 26 20:12:48 <rjk2> is that with one card?
Dec 26 20:12:51 <luke-jr> yes
Dec 26 20:13:04 <rjk2> i built myself a mini-itx system, jsut to do that
Dec 26 20:13:04 <luke-jr> also, OpenCL is a memory hog :/
Dec 26 20:13:37 <luke-jr> rjk2: note that PCI bus, and probably PCI-E extenders, will be all-or-nothing to VMs
Dec 26 20:13:49 <luke-jr> ie, you might be unable to isolate single cards in separate VMs

And, the issue requiring a restart that Luke-jr is discussing was worked around later in a different chat (put the machine to sleep, and bring it back out, instead of rebooting).

[ice_chill]

Any update ? This was the most interesting mining non-FPGA thread.

Is the Rig all running now ?

[rjk]

Any update ? This was the most interesting mining non-FPGA thread.

Is the Rig all running now ?

Not really, I just got the fan controller but haven't had a chance to install it. I'd like to do it tomorrow, but that's the day of the state auction, and I sure as heck am not going to miss that. http://www.das.ohio.gov/Divisions/GeneralServices/Surplus/WarehouseNextAuction.aspx

I also only have 13 cards, so I need to get some more. I guess I'm kind of waffling around, hoping that a pile of 7990s drop in my lap sometime soon, although that is rather unlikely lol.

[Garr255]

Ahh I really wish I could go to that. Piles of computers!!

[1l1l11ll1l]

I work just down the street from our statue university. Their surplus outlet is always packed with equipment. 3 story building, 1st floor is almost completely dedicated to computers and servers.

[Garr255]

How much does it cost to walk around and pick through everything? And do you know if they have any old computers? I could always use another c64

[twoBitBasher]

Not really, I just got the fan controller but haven't had a chance to install it. I'd like to do it tomorrow, but that's the day of the state auction, and I sure as heck am not going to miss that. http://www.das.ohio.gov/Divisions/GeneralServices/Surplus/WarehouseNextAuction.aspx

Mmmm serrrrrvers

[rjk]

How much does it cost to walk around and pick through everything? And do you know if they have any old computers? I could always use another c64

How about an IBM Selectric II?