There is still a risk of a chain split even with a backward compatible change such as BIP16 or BIP17. To avoid that from happening, Gavin is trying to get miners to commit to supporting one of the BIPs up front and then coordinate the timing for them to actually switch over. But lets say he gets the commitment, sets a date for the switch over, but for one reason or another, only 30% of the miners start enforcing the new rules. Let's also assume we have a miner that's not restricting itself to transactions that are considered "standard" (*ahem* Luke-jr*). In this scenario, you could see a transaction that is considered valid under the old rules (may not be standard, but could be perfectly valid), but which is invalid under the new rules (see one of the methods of spending a BIP16 or BIP17 transaction under the old rules without authorization). You now have a chain split because the old chain still has 70% of the hash power and the new chain won't be able to outrun the old chain.
Now, to be clear, I don't expect this to happen with the BIP16/17 transition as I expect the miners will all do what they're expected to do and pretty much all transition to the new rules. But it does rely on people acting in a coordinated fashion.
So, here's how I think it could work in the future that would work even if miner coordination is botched. Clients should be designed to support 2 script engines and their associated block chains. The script engine could be factored out as a shared lib or some plugin that could be installed or swapped out independent of the client itself. Normally, people would just have one script engine installed and the client would just manage a single block chain. But when a script change is being implemented. A new script engine could be distributed. People would install this second script engine. When two engines are installed, the client would keep track of two block chains and would validate all transactions against both block chains and refuse to accept or propagate transactions that don't successfully validate against both engines. If the script changes are backward compatible, there is a good chance there won't be a block split even though there are two script engines, so the client is just managing the single block chain as usual (but still validating all transactions and blocks against both engines). Miners can then decide on a schedule to start performing validation using the new script engine and attempt to make the transition in a coordinated fashion. But the good news for everyone else is that if the miners screw up, they only hurt themselves and not everyone else. Some of the miners might be creating blocks with worthless coinbase transactions, but that's their fault for not getting their act together. Miners will have a very strong incentive to all be mining on one chain or the other…so any division of mining power between two block chains should be very temporary (it should only take a few days at most for the super majority of miners to get on the chain that appears like it will be the survivor). Once it becomes clear that one of the engines/chains is going to survive (this might be indicated by blocks appearing at very long intervals or the difficulty plummeting), people can drop the dead chain.
I think this (or something close to it) would enable smooth transitions in the future, even if the changes would lead to a chain split. The only requirement would be that the proposed new chain still support old style transactions (perhaps the new engine could reject long since deprecated transaction styles in order to phase out the use of them, but transaction styles still in widespread use would need to be supported). In fact, a chain split might actually be preferable in this case because you would know definitively which set of rules are being enforced by the majority of miners (with the proposals for BIP16/17, assuming a split doesn't occur, you don't have a reliable way to know for certain if the majority of miners are actually enforcing the new BIP16/17 rules).
You could distribute a new script engine months ahead of a proposed date for miners to switch over. That would give ordinary users a fair amount of time to install the new engine and prepare for the change (especially if the change will cause a chain split).
* I actually think it's good that we have a pool that accepts all valid transactions…I would be more nervous if no one accepted strange transactions and no one was thinking about the implications…if someone can do something that might compromise bitcoin, someone should do it (as a white-hat kind of exercise).