The driver is absolutely wedged into the operating system kernel and unremovable. Getting the picture yet?
I do understand what you say, but I was able to unload it when it is not attached to cgminer, and then load it again. So, it becomes unremovable only when one of the cards gets sick and/or dies.