i got the same problem.
seemed to me it was like loading the driver 2 times, and getting a bluescreen at the second time.
i have not found a solution but reinstalled windows.
here is what was: i got a 4350 and just inserted the new 5850, installed drivers (did not remove old ones). during install, i installed some other stuff too and some component from ati couldnt be installed due to msiexec busy. so i installed the drivers again (no reboot again, no deinstall)
after a reboot i had bluescreens.
with vga mode enabled, i have no bluescreens and no 800x600 vga but full resolution. funny its working as it should with vga enabled so wtf.
i pulled a full disk image backup and removed the 4350 driver before installing 1 time the new driver. even with drivercleaner. same thing.