Can we call it a perfect transition from P2MS since P2MS is limited to just 3 public keys and will require 2 of 3 to spend an UXTO.
AFAIK there are only standard rules that are limiting P2MS compared to P2SH (multisig) otherwise the only limiting factor in any MultiSig scripts is MAX_PUBKEYS_PER_MULTISIG (=20) and the OP/SigOp count as far the consensus rules are concerned.
whereas P2SH gives room for custom redeem script
Technically you can pay to any custom script (in your scriptpub). The benefit of P2SH is that it uses the hash of the script so you don't need to reveal that script and also it helps you create an address (something you can't do in P2MS or any custom output script) and give that address to others to pay you.
this brings me to my question if P2SH is the most prominent or is there a new BIP that has better qualities?
P2SH is one of the early changes to the consensus rules that involved the bitcoin scripts, I don't know if it is the most prominent though.
due to bug on P2SH multi-signature
That is not a bug in P2SH, it is a bug in OP_CHECKMULTISIG(VERIFY)
I also don't think it was a bug but an intentional "feature" to have an open room for future soft forks where the dummy item is used for something while still being backward compatible.