rjk (OP)
Sr. Member
Offline
Activity: 448
Merit: 250
1ngldh
|
|
May 07, 2012, 09:41:09 PM |
|
Have you enabled IOMMU in the BIOS? It is my understanding that is necessary for PCIe pass-thru.
IOMMU is for AMD; I have an Intel platform so I have VT-d. And yes, it is enabled. PVE kernel panics when I try to hotplug the devices to the virtual machine, and the virtual machine won't boot when the devices are added to the config file when it isn't booted.
|
|
|
|
rjk (OP)
Sr. Member
Offline
Activity: 448
Merit: 250
1ngldh
|
|
May 07, 2012, 09:50:31 PM |
|
have you tried ESX(i)? i think it may be a little bit more user friendly, vCenter management software, just install it to an USB stick, and you should be able to create VMS (if you have some storage repository), after you have done that, you can (try to) add devices with the configuration manager (vcenter), you can do that from the start but i always check if it is working the normal way...
why did you choose PVE? you feel more comfortable with it?
It was a recommendation, because it is based on a pre-setup KVM kernel. I heard ESXi has a limitation of 4 devices though?
|
|
|
|
DiabloD3
Legendary
Offline
Activity: 1162
Merit: 1000
DiabloMiner author
|
|
May 07, 2012, 10:05:52 PM |
|
Have you enabled IOMMU in the BIOS? It is my understanding that is necessary for PCIe pass-thru.
IOMMU is for AMD; I have an Intel platform so I have VT-d. And yes, it is enabled. PVE kernel panics when I try to hotplug the devices to the virtual machine, and the virtual machine won't boot when the devices are added to the config file when it isn't booted. No. IOMMU is a industry term for it. VT-d is an implementation of IOMMU, AMD, Sun, and IBM also offer implementations of it.
|
|
|
|
DiabloD3
Legendary
Offline
Activity: 1162
Merit: 1000
DiabloMiner author
|
|
May 07, 2012, 10:19:15 PM |
|
AMD has published a specification for IOMMU technology in the HyperTransport architecture.[1] Intel has published a specification for IOMMU technology as Virtualization Technology for Directed I/O, abbreviated VT-d.[2] Information about the Sun IOMMU has been published in the Device Virtual Memory Access (DVMA) section of the Solaris Developer Connection.[3] The IBM Translation Control Entry (TCE) has been described in a document entitled Logical Partition Security in the IBM eServer pSeries 690.[4] The PCI-SIG has relevant work under the terms I/O Virtualization (IOV)[5] and Address Translation Services (ATS). So, are we done playing wikipedia lawyer?
|
|
|
|
rjk (OP)
Sr. Member
Offline
Activity: 448
Merit: 250
1ngldh
|
|
May 07, 2012, 11:15:21 PM |
|
Anything new to report? If you get a kernel panic on a linux guest during boot or just can't get a linux guest to boot, I'd be interested in knowing if a similarly configured windows VM does the same.I think we had figured out about the device id's for your devices last.
It was the host that panicked, not the guest. The output is on the previous page. Maybe ill try a windoz guest and see what happens.
|
|
|
|
rjk (OP)
Sr. Member
Offline
Activity: 448
Merit: 250
1ngldh
|
|
May 08, 2012, 01:28:47 AM |
|
I'm on my phone so it makes it a little hard to elaborate on all the points, but I'll try to remember all of them. Yes there is onboard video, it doesn't show in that list because - grepped for "Cypress XT" not "VGA". When I grepped for VGA, I got several screens of output and didn't sift through it thoroughly. I have tried hotplugging and coldplugging different devices. Yes, I would assume that the host is trying to use one of the cards and that could cause a conflict. I just had a thought - would it be possible that the host is glomming on all the cards just because they are there? I should change my BIOS to boot the internal video first and use it as primary display. When hotplugging, the host (dom0) was what panicked. When coldplugging, the guest (domU) either flashed the VNC terminal on and off repeatedly (when a single device was configured) or showed a black screen and froze (no networking or anything) when all cards were assigned. Unfortunately development is kind of on hold because of time constraints (weekender) and because these fans at 100% are damn annoying. I need to get the following list of stuff done: Custom bracket - need to find a small fabricator near me Copper bus bar - determine correct size, layout, and method of attaching conductors PSU - figure out how to turn it on PCIe 6/8 pin connectors - obtain a bunch and figure out how to hook them to the bus bars (ring/spade terminals? solder? clamp?) Air channel/duct - needed so that the air doesn't escape past the heatsinks over the top - also a job for the fabricator - needs to be slotted so the upper level PCIe extenders can pass thru Fan contoroller - wire it in once it arrives and set up the control software (to be written, coder is lined up for the job) Damn, I wish it was as easy as throwing some money at it and putting it together like Legos. Problem is, I'm kinda weak when it comes to the software and scripting/programming side of things. Once it's done, does anyone wanna buy it and have the distinction of owning the worlds fastest, most complicated GPU based rig? It will need a 7KW power outlet and cooling.
|
|
|
|
rjk (OP)
Sr. Member
Offline
Activity: 448
Merit: 250
1ngldh
|
|
May 08, 2012, 01:42:59 AM |
|
one quick note, after what you said i remember you have to unbind the pci device first (logical right), have you done that? i cant recall having read it(but that doesnt say too much) 4. unbind device from host kernel driver (example PCI device 01:00.0) Load the PCI Stub Driver if it is compiled as a module modprobe pci_stub lspci -n locate the entry for device 01:00.0 and note down the vendor & device ID 8086:10b9 ... 01:00.0 0200: 8086:10b9 (rev 06) ... echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVMin xen it was something with pciback http://wiki.xensource.com/xenwiki/Assign_hardware_to_DomU_with_PCIBack_as_modulereally not sure though.. Having to unbind it from the host first makes perfect sense, I didn't really think of that. I'll have to try those steps and see how I get on.
|
|
|
|
rjk (OP)
Sr. Member
Offline
Activity: 448
Merit: 250
1ngldh
|
|
May 08, 2012, 10:19:58 PM |
|
Having to unbind it from the host first makes perfect sense, I didn't really think of that. I'll have to try those steps and see how I get on.
Played with it a little more. Unbinding and rebinding to PCI_stub seemed to work (Is this persistent? If not, how do I make it save these settings between reboots?), and I was able to attach a device to a VM. However, X segfaulted with error 11. I was able to lspci and see the video card from inside the VM though. PVE doesn't seem to like having more than 2 PCI devices per VM, and if you add more to the VM config, it simply removes them for you. I tried installing a new VM with the cards pre-attached, and the bootdisk just showed a blank screen and went no further. I tried installing Windows 7 x64, but it kept asking for a driver for the CDROM which I didn't have or know where to get.
|
|
|
|
rjk (OP)
Sr. Member
Offline
Activity: 448
Merit: 250
1ngldh
|
|
May 08, 2012, 10:55:01 PM |
|
What's the smallest Linux install that I can have that will mine at full speed, and how little RAM can I assign to it? If I have to have 2 devices per VM, that would mean 17 VMs in a dual-GPU situation, which would suck. And, since I have only 8GB of RAM, I will have to have fewer than 512MB RAM per VM. I don't know how much RAM PVE uses for Dom0, but if I assume 1GB (like XenServer), that leaves me with 7GB divided by 17 VMs, equals ~420MB RAM per VM. I suppose that should be enough for something that is stripped down, but I doubt it would run BAMT with all its monitoring and stuff.
|
|
|
|
Gomeler
|
|
May 09, 2012, 01:06:38 AM |
|
LinuxCoin maybe? although it doesn't say how much RAM you need http://www.linuxcoin.co.uk/wiki/index.php/Headless_LinuxcoinThe disk image contains the AMD APP SDK, ATI Catalyst & Drivers, poclbm & phoenix miner, Bitcoin, and more. the Xeon E3-1225 seems to have just 4 cores & 4 threads .. and i think the host wants one , so maximum 3 virtual machines possible, each getting a thread? You can overprovision the CPU on a VM host. So long as you don't peg the CPU cores on all your VMs, the hypervisor will load balance your CPUs to provide enough CPU power for your VMs. The major obstacle is always ram and storage IOPs. Luckily ram is dirt cheap.
|
|
|
|
Gomeler
|
|
May 09, 2012, 07:27:41 AM |
|
If ram becomes an issue I'd suggest just trying 8 GB dimms. It may work, I've seen more bizarre things than that.
|
|
|
|
terrytibbs
|
|
May 09, 2012, 02:37:29 PM |
|
luke-jr is virtualizing gpu's with kvm. I'll try to dig up that info or have a conversation with him or see how he likes this thread.
Just don't mention religion.
|
|
|
|
rjk (OP)
Sr. Member
Offline
Activity: 448
Merit: 250
1ngldh
|
|
May 09, 2012, 02:37:54 PM Last edit: May 09, 2012, 02:51:27 PM by rjk |
|
luke-jr is virtualizing gpu's with kvm. I'll try to dig up that info or have a conversation with him or see how he likes this thread.
You know, I did discuss it with him months ago, but that was only the most basic bare minimum of information, because I didn't have compatible hardware. I'll need to grep my IRC logs to see if he gave out any technical information. All I remember was that it was 1 GPU and that it kept crashing. EDIT: I found some stuff, and I know there is more, but I can't be bothered to find it all right now. Dec 26 19:54:12 <rjk2> gmaxwell: i thought there was a driver limit of 8 gpus maximum Dec 26 19:54:28 <gmaxwell> Driver. Yup. Dec 26 19:54:35 <gmaxwell> So run multiple copies of the driver. :) Dec 26 19:54:55 <gmaxwell> (according to luke the ati drivers work in kvm with the pci remapping stuff) Dec 26 19:55:35 <gmaxwell> so in _theory_ you could start two VMs, map 4 cards to each and run 16 GPUs on a system. Dec 26 19:55:45 <gmaxwell> (presuming you could plug in that many) Dec 26 19:55:58 <gmaxwell> (and presuming your motherboard doesn't catch fire) Dec 26 19:57:38 <rjk2> I have been trying to find someone that has actually been able to mine successfully in a virtualized environment Dec 26 19:57:48 <gmaxwell> luke-jr: Dec 26 19:57:52 <rjk2> i know it ought to work Dec 26 19:58:16 <luke-jr> gmaxwell: only with motherboards/CPUs that support it Dec 26 19:58:44 <rjk2> vt-d, and iommu right Dec 26 19:58:49 <luke-jr> VT-d in my case Dec 26 19:58:54 <rjk2> but does it work well? Dec 26 19:58:55 <rjk2> stable? Dec 26 19:58:58 <luke-jr> also, Intel pulled a big scam earlier this year Dec 26 19:59:02 <rjk2> because i want to try it Dec 26 19:59:09 <luke-jr> well, my Radeon still manages to crash my host system sometimes Dec 26 19:59:17 <rjk2> hmm Dec 26 19:59:25 <gmaxwell> rjk2: you get diminishing returns packing more cards onto one system. ::shrugs:: Dec 26 19:59:26 <luke-jr> usually when I try to reinitialize it Dec 26 19:59:30 <luke-jr> which is mostly playing around Dec 26 19:59:38 <luke-jr> if I knew a proper way to do that, it might be OK Dec 26 20:00:09 <rjk2> what i wanted to do was use intel vt-d with built-in video powering my desktop environment, and discrete cards mining Dec 26 20:00:23 <luke-jr> rjk2: tht's what I do Dec 26 20:00:24 <rjk2> z68 based Dec 26 20:00:46 <rjk2> luke-jr: sounds good, using kvm or xen? Dec 26 20:01:21 <luke-jr> KVM Dec 26 20:01:47 <luke-jr> Z68 = no VT-d Dec 26 20:01:53 <rjk2> wait what Dec 26 20:02:05 * catalase_ is now known as catalase Dec 26 20:02:08 <rjk2> i know the K procs with overclocking don't have it Dec 26 20:02:17 <rjk2> but z68 doesn't either? Dec 26 20:02:20 <luke-jr> nope Dec 26 20:02:26 <luke-jr> http://ark.intel.com/compare/52816,52812 Dec 26 20:02:31 <rjk2> damn it Dec 26 20:02:37 <luke-jr> return it :P Dec 26 20:03:13 <rjk2> is that what "embedded options available" means? VT-d? Dec 26 20:03:19 <luke-jr> no Dec 26 20:03:26 <luke-jr> Intel® Virtualization Technology for Directed I/O (VT-d) Dec 26 20:03:37 <luke-jr> it's exclusive to Q67 Dec 26 20:03:38 <rjk2> oh, i see its down lower Dec 26 20:03:41 <luke-jr> at least for Sandy Bridge Dec 26 20:03:45 <rjk2> hmm Dec 26 20:04:01 <luke-jr> originally it wasn't Dec 26 20:04:03 <rjk2> man that is lame Dec 26 20:04:08 <luke-jr> but Intel bugged, and decided the solution was to fix their specs Dec 26 20:04:20 <luke-jr> ie, they said the bug was the advertising VT-d on the others Dec 26 20:04:31 <gmaxwell> doh! that sucks. Dec 26 20:04:34 <luke-jr> yep Dec 26 20:04:38 <luke-jr> but they also bugged the SATA Dec 26 20:04:46 <luke-jr> so I returned my H67 and got a Q67 Dec 26 20:04:48 <luke-jr> :D Dec 26 20:04:53 <rjk2> lol too true >.> Dec 26 20:04:58 <luke-jr> though it was VERY difficult to get Q67 at the time Dec 26 20:07:34 <rjk2> luke-jr: have you had the opportunity to test IOMMU (AMD) systems in the same manner yet? Dec 26 20:08:21 <luke-jr> no Dec 26 20:08:36 <luke-jr> I think someone did though, and found they had to do 1 VM per card Dec 26 20:08:42 <rjk2> lolwut Dec 26 20:08:44 <luke-jr> I've only tested 1 on VT-d Dec 26 20:09:18 <luke-jr> the annoying thing is, if the card crashes, I pretty much have to reboot Dec 26 20:09:33 <luke-jr> in theory, I *should* be able to 'remove' the card and power cycle just it Dec 26 20:09:39 <luke-jr> but I haven't figured out how in practice Dec 26 20:09:51 <luke-jr> even if I remove and rescan, it doesn't get reset Dec 26 20:10:09 <luke-jr> upgrading fglrx *might* help Dec 26 20:10:28 <luke-jr> since newer versions support hot-switching between IGP and discrete on laptops Dec 26 20:10:58 <luke-jr> (the error I was getting found only Google results talking about that) Dec 26 20:11:37 <luke-jr> latest fglrx *seems* to be working just as well as the good old 2.1 now tho Dec 26 20:12:19 <luke-jr> ie, no CPU hogging, and full 308 MH/s Dec 26 20:12:48 <luke-jr> I don't have a sane way to reproduce the card crashing though, so no idea how many months until I can confirm it reinitializes Dec 26 20:12:48 <rjk2> is that with one card? Dec 26 20:12:51 <luke-jr> yes Dec 26 20:13:04 <rjk2> i built myself a mini-itx system, jsut to do that Dec 26 20:13:04 <luke-jr> also, OpenCL is a memory hog :/ Dec 26 20:13:37 <luke-jr> rjk2: note that PCI bus, and probably PCI-E extenders, will be all-or-nothing to VMs Dec 26 20:13:49 <luke-jr> ie, you might be unable to isolate single cards in separate VMs And, the issue requiring a restart that Luke-jr is discussing was worked around later in a different chat (put the machine to sleep, and bring it back out, instead of rebooting).
|
|
|
|
ice_chill
|
|
May 18, 2012, 11:12:04 PM |
|
Any update ? This was the most interesting mining non-FPGA thread.
Is the Rig all running now ?
|
|
|
|
rjk (OP)
Sr. Member
Offline
Activity: 448
Merit: 250
1ngldh
|
|
May 19, 2012, 01:20:05 AM |
|
Any update ? This was the most interesting mining non-FPGA thread.
Is the Rig all running now ?
Not really, I just got the fan controller but haven't had a chance to install it. I'd like to do it tomorrow, but that's the day of the state auction, and I sure as heck am not going to miss that. http://www.das.ohio.gov/Divisions/GeneralServices/Surplus/WarehouseNextAuction.aspx I also only have 13 cards, so I need to get some more. I guess I'm kind of waffling around, hoping that a pile of 7990s drop in my lap sometime soon, although that is rather unlikely lol.
|
|
|
|
Garr255
Legendary
Offline
Activity: 938
Merit: 1000
What's a GPU?
|
|
May 19, 2012, 02:22:23 AM |
|
Ahh I really wish I could go to that. Piles of computers!!
|
“First they ignore you, then they laugh at you, then they fight you, then you win.” -- Mahatma Gandhi
Average time between signing on to bitcointalk: Two weeks. Please don't expect responses any faster than that!
|
|
|
1l1l11ll1l
Legendary
Offline
Activity: 1274
Merit: 1000
|
|
May 19, 2012, 03:31:03 AM |
|
I work just down the street from our statue university. Their surplus outlet is always packed with equipment. 3 story building, 1st floor is almost completely dedicated to computers and servers.
|
|
|
|
Garr255
Legendary
Offline
Activity: 938
Merit: 1000
What's a GPU?
|
|
May 19, 2012, 04:04:47 AM |
|
How much does it cost to walk around and pick through everything? And do you know if they have any old computers? I could always use another c64
|
“First they ignore you, then they laugh at you, then they fight you, then you win.” -- Mahatma Gandhi
Average time between signing on to bitcointalk: Two weeks. Please don't expect responses any faster than that!
|
|
|
twoBitBasher
Member
Offline
Activity: 85
Merit: 10
|
|
May 19, 2012, 10:47:47 AM |
|
Mmmm serrrrrvers
|
If you think my comments have benefitted you it would be nice to hear thanks Doge: DMnfgNp1HQSjtTZ1HcWiYtMwoGP5xcYDcz
|
|
|
rjk (OP)
Sr. Member
Offline
Activity: 448
Merit: 250
1ngldh
|
|
May 19, 2012, 07:05:44 PM |
|
How much does it cost to walk around and pick through everything? And do you know if they have any old computers? I could always use another c64 How about an IBM Selectric II?
|
|
|
|
|