A simple limit on orphan blocks will solve this. Since they're not propagated and the attack is on memory, it seems pretty little to worry about. You could handle millions of such blocks with no problem.
Not really: MAX_BLOCK_SIZE is 1,000,000, so I could make megabyte-sized blocks and send them your way. Every thousand of these I send you chew up 1GB of memory on your end, enough to make anyone sad (and perhaps even crash the client due to a process limit). To go further, if I send you a million such blocks, that'll eat up 1 TB of swap space, which would bring down almost any normal user's machine, and maybe even some miners (admittedly, sending you 1TB takes a while, but if you're a server machine and no one's watching, then who cares?).
Of course, you could just restart the client and be in the clear, but it seems pretty easy to defend against this kind of attack. As you said, limit the number of orphan blocks you're willing to keep in memory. Even better, when choosing which orphans to forget about, preferentially forget those that are linked to a previous orphan that you know about. That way, you'll slowly build back to a genuine fork point, if that's what's going on. But to do this properly, as soon as an orphan chain is linked into the main chain, you'd need to push a request for all the blocks following the tip of that orphan chain, since you may have received them before and chosen to forget about them.