I went through this problem myself a few days ago. When installing 1 GTX 460 on primary or secondary PCI-e on my motherboard (asrock 870 extreme3), GUI miner wouldn't recognize my HD 6970 or OpenCL support - the only thing that worked in any combination of PCI-e slots was CUDA. After 3 days of testing different setups I finally found a solution that works (for me at least), but you could give it a try. Also note that this was only tested on Windows 7 64 and GUI miner.
1- uninstall all drivers/softwares related to your GPUs and do a driver cleaner (I use driver sweeper)
2- plug in all you graphic cards on the motherboard, in desired PCI-e configuration (example: 5770 on the first slot, 5830 on second, and nvidia on third)
3- power up you computer and let the operating system install all generic drivers for the adapters (you may want to disconnect from internet while doing this, cause Windows 7 tends to download driver software for your graphic adapters on the fly and install them in a weird order)
4- after step 3 is completed, install the latest driver package for you AMD cards - reboot the computer
5- after reboot, start GUI miner and make a new OpenCL miner - it should recognize your AMD GPUs and CPU. Select desired adapters and save the them on exit. No reboot required for next step.
6- Install the driver package for nvidia card and now reboot.
7- after reboot you might need to reconnect to internet if you followed step 3 exactly. Now open GUI miner and make a new CUDA miner - save it and start the CUDA miner to test that it works. If everything is OK you can stop it and close the miner.
8- This step might be a bit trickier so I'll tell you a short story. In my case, when installing nvidia drivers over the AMD software my GTX 460 would completely brake OpenCL suport and replace it with CUDA. When trying to mine with my HD 6970, GUI miner would report something like "nvidia CUDA - starting", but it wouldn't start at all (and that's to be expected, since I was mining on a AMD GPU and GUI miner was trying to connect with CUDA). When testing my second GPU (GTX 460) everything was OK and mining started in a mater of seconds. Disabling CUDA from nvidia control panel won't help, cause GUI miner will still try to connect to it. So my workaround is a bit messy , but it works for me.
8-1. in GUI miner folder make a another copy guiminer.exe (name it anything you like - mine is called guiminer - 2.exe
8-2. open the first executable of GUI miner and make sure is is configured for the AMD cards - close and save. Now open the second executable and configure this one for nvidia - close and save.
8-3. go to Computer -> Properties -> Advanced system settings -> Environment Variables. On the System variables tab we'll make a new variable, so click on NEW and name it "CUDA_VISIBLE_DEVICES" with a variable value of "9" - Click OK on all windows. The 9 value basically tells GUI miner that only device #9 is CUDA capable, and wont try and search further for any other devices. Since device 9 dose not exists GUI miner wont be able to connect with CUDA, so it's switching to AMD OpenCL instead.
8-4. Open the first executable of GUI miner and start mining with you AMD cards - After that let it do his stuff.
8-5. Now, to mine with your nvidia card simply go back to Environment Variables, select the "CUDA_VISIBLE_DEVICES" variable and change the value to that of your CUDA capable device (like "3", if your PCI-e slot numbering starts from 1, or "2" if numbering starts from 0). In my case the devices are from 0 to 1, so the value should be and is "1". Save the variable.
8-6. Open the second executable of GUI miner (in my case guiminer - 2.exe) and start the mining on the nvidia card. If everything is OK it should connect to CUDA and start mining.
I cannot guarantee that this workaround will work for everyone, but in my case it dose. Also, every time you stop your AMD mine or restart the PC you must go through steps 8-2 to 8-6 again, or the miners wont work on AMD GPUs.
Hope this helps someone.