There are some scary code in the reference implementation
What "scary code" are you referring to?
This is incorrect Versions prior to 0.8 were inconsistent with _themselves_: Block verification was non-deterministic for some large blocks, with acceptance depending on the precise layout on disk of some database data structures. The fork was triggered by a miner that changed their settings to produce larger blocks than typical, and would have happened without any new versions in play. The precise nature of the issue was initially misunderstood as being 0.8 vs before, because all the 0.8 were on one side, but in fact most of the split was pre-0.8 vs pre-0.8.
This may or may not have much relevance to your thinking, but please get your facts straight. It's irritating to see this misinformation/misunderstanding continually repeated.
Let's say ~50% of the mining capacity uses the reference implementation and ~50% uses another implementation.
It's not what miners are using that matters. If your are mining and your blocks are being rejected by the user's systems because they fail validation then you're not actually mining, regardless of how much hashrate you have.
Suppose a big change was actually made, there’s nothing forcing miners to take the update. What are some of the risks of having miners on the network using different versions of the protocol? Not necessarily malicious clients, just outdated ones?
(since you appear to be asking about intentional changes:)
When changes are adopted to the blockchain rules they're made in a way which is intentionally backwards compatible with old versions, called a "soft fork". This is accomplished by constructing the change so that it strictly narrows the set of permissible blocks. Because nothing invalid became valid, old nodes also accept these blocks. To avoid issues where old nodes would create blocks that get forked off, an effort is made to only narrow the validity of blocks in ways that an old node wouldn't have constructed an invalid block, and the new rules are only activated when a strong super-majority of miners has signaled an intent to enforce them.
Keep in mind that the vast majority of all conceivable changes are actually bad and open up new attacks: Most things people post about (myself included) turn out to be bad ideas on further reflection. The 'mitigate 51% attacks' link you provided is, I think, one such example: that approach fails to prevent any interesting attacks (attacker can easily meet the criteria by including many of the prior transactions) and also opens up new attacks where none existed before (by intentionally preparing two forks and broadcasting to half the network simultaneously the network can be cheaply split and work against itself; this is a general flaw pattern in approaches that make block preference depend on multiple strong tie-breakers).
