well i had some stability problems, at first I thought it was virtual memory related, but it was actually oc settings, i'm trying to stabilize miner with 32gb pagefile and lower core oc, memory -502 lowers hashrate to 11,6mh.sec, while +500 has almost zero effect, so it's better to keep at 0, with c29 oc settings i have 12,6mh/sec, i will try to maximize stability with 0 memory oc and highest stable core oc on every gpu, well without using ohgod eth enlargment pill i lost around 1mh.sec of total hashrate, actually it's almost 1.3mh.sec
well with 3XGTX1080 EVGA FTW i put +50 core (using second bios) and it operates around 1750MHZ, 1XGTX 1080 MSI gamer x i put +180 core and it operates around 1650MHZ, 1XGTX1080 Ti zotac amp i put +120 MHZ and it works around 1550MHZ, all 5 gpu are on TDP 70% and memory 0 + pill, here's my first result (more or less stable):

https://i.imgur.com/hwrh9hG.png

i will test it in longer period and see how it will do on nicehash pool as well, actually i find out that with pill even setting memory at -502 gives you same hashrate as with 0, without any core oc i got only 13.3mh/sec, with core oc i got around 13.6mh/sec
well using this setting also increases hashrates (memory -502 + pill):

https://i.imgur.com/FlQBWsA.png

i will test it for stability and report back, i think it can operate around 13.8+mh/sec without losing stability (you have to lower one of the GPU core setting (the one that uses teamviewer) otherwise miner will crash with very high core clock

i managed to stabilize at around 13.7mh/sec :
3XGTX1080 EVGA FTW core +0 (using second bios which has automatic core oc)
1XGTX1080 MSI gamer x Core +160
1XGTX 1080Ti Zotac amp core +120

all 5 GPU memory -502 TDP 70% + pill + force p2 state off, after few hours of testing i will report actual nicehash pool hashrate and also check stability on those settings:

https://i.imgur.com/movKvh7.png