What do you mean with: "the wallet doesn't scale all that well".
For instance, it has to be held in RAM. Right now there's no way (without modifying the code a lot) to hold a huge wallet in a database.
Even if there was, the wallet code is written for simplicity and correctness, not speed. It has a bunch of quadratic and worse algorithms in it. So if you have a giant wallet creating spends would take a long time, things like that.
Nothing that can't be fixed, it's just not been a priority.