So far as the rest of the world is concerned, each address you generate is completely unique. If you create two addresses, there's no way for anyone to connect the two of them (unless you transfer money between them a bunch of times).
There's no reason to swap wallets, and you run the risk of messing it up and losing coins...
However there is currently also no way to control which address any payment you make comes from. So you could be unwittingly connecting "Grandmas birthday money" to "Hot goat porn". Which could lead to some loss of anonymity.
But no, you cannot currently hot swap wallets.