Quick question, the 24 word phrase is only needed if you forget your passcode?
By passcode do you mean the 4-8 digit PIN you use to unlock your Ledger? If you forget your PIN then yes, you will need to use your seed phrase to recover access to your wallets. You will also need your seed phrase if anything happens to your Ledger device, such as it is lost, stolen, damaged, or simply wears out after years of use. But as long as your Ledger is working fine and you know your PIN, then you do not need to use your seed phrase in usual day-to-day operations.
If I was to lose my ledger, I can just buy a new one but I must have my 24 words to access it again?
Correct. Your PIN number is specific to the device and simply unlocks the device. It has no relationship whatsoever to your actual wallets, private keys, or coins. Those are backed up exclusively by your 24 word seed phrase, plus any additional passphrase you may or may not have set. If you lose your Ledger, you can restore your seed phrase on any wallet which supports BIP39 seed phrases, which is almost every major software or hardware wallet, although doing so on a fresh hardware wallet will be the safest route.