It should be able to boot at least.
Power cables are plugged into both cards right?
If the 9800GT boots by itself, and you are getting rid of it, why do you need to make sure it boots with the radeon?
You have a bootable CD, like a WindowsPE disc or UbuntuLive disc? See if that works. Could be a power issue. It might be 500watts, which should be enough to boot at least, but could be an amp issue maybe? Does the PSU have different rails, and each GPU on a different rail?
Suggestions might seem silly, but sometimes something obvious could and can usually be overlooked.
Hey thanks for pointing that out. I thought it only had 1 rail but I looked it up and it has two. Need to look inside again, seems like it was balanced tho since I wasn't using any 4 pins and I was using them for my 9800gt and the 2 6 pins for my 5850. Will try a different way maybe with more power from booster psu.
Also for some weird reason I had problems installing windows when I had my 5850 installed, but it worked with my 9800gt fine.
And the power supply is only a few years old, seemed like I was buying a quality one..80plus bronze and its still a common one on newegg.