They don't even need a Ledger to steal your coins if they have access to your 24 word seed. They can import it in any software wallet that supports the format and launch the wallet from there.
I don't have a Ledger myself, but the 4 digit security code you are talking about, did you set it yourself during the setup process? If so, that infromation will be lost when you enter the seed into a new wallet, thus allowing spending.
I can confirm your suspicions. The 4 digit security code, nor the security card is needed to restore your wallet. The pin number is only needed to open your wallet, and can be choses by yourself, the security card is linked to the hardware wallet itself, but only the 24 words are needed in order to generate the xpriv.
In other words:
-if somebody steals your ledger, but they don't have the pin nor the security card: you're safe.
-If somebody steals your ledger, and your security card, you're prone to brute-forcing.
-if somebody steals your ledger, sees your pin but doesn't steal your security card: you're safe.
-if somebody steals the paper with the 24 words: they have access to all your funds, without having to brute-force anything.