Hmm 1080TI have 8GB so i dont think that this is problem with increasing DAG file size in ethereum. (by any chanse
But you could try running other miners like teamredminer or phoenixminer.
also if you are using some extra parameters in etherminer you could try removing them or if you are using ohgodapill, you could try without it.
Also try lowering OC, might be the cause also.
If you will still have problems then you could just run our simplemining.net operating in which you do not need to mess with the drivers or system settings.
http://bitcointalk.org/index.php?topic=1541084