Everyone has told me HBCC requires 8GB system Ram and about 8 GIG of page file per card. I am running 6 Vega 56's. I spent a week tweaking it all.
Also running mixed cards could be your issue. Cast XMR works great for Vega, but not 100% stable with other cards as stated in V.06 Rx cards are still experimental.
Try running it with just the vega's, then just the RX's see if it makes a difference for stability, but to get the high hash rates, HBCC will need the 8 gig system ram.
Anyone else have any Mixed AMD GPU Experiences?