Ok. I have finally gotten around to setting up tomato on my wrt54gl router. First off, tomato is amazing!
Secondly, I have cgminer now running on every one of my rigs!
It looks like the culprit was the total number of open connections. Tomato is reporting I have between 4800 and 5200 tracked connections at any point in time.
I'll be doing some more tweaking over the weekend and we'll see where we get to. For now though, it's tomato FTW!