The question I really want to ask is can a colored coin by ruined by an outsider sending unanticipated coins to its address? And if so is the colored coin concept fatally flawed?
A valid coin is simply an unspent output from a past transaction. Every coin has an associated script, and in order to spend the coin, you must provide the script with an input such that it evaluates to 'true'. The most common example is where the script contains a hash of a public key, i.e. a bitcoin 'address', and the input you must provide is the full public key and a signature from the corresponding private key. Here it is in detail:
https://en.bitcoin.it/wiki/Script#Standard_Transaction_to_Bitcoin_address. Different coins can have the same script, or 'address', and the blockchain still sees them as distinct. An "outsider" can send me a coin with the same script as one of my own, but the only way these coins can be combined, from the blockchain's point of view, is if I spend them together in a transaction later on. If my coin is colored, then I obviously wouldn't do this on purpose if it would ruin the coin's coloring.