it depends on the output type that is being spent in your inputs!
each output has a locking script that requires an unlocking script in your input when you spend it. when the requirement for "unlocking" takes more space, the input can add more size to your final transaction.
for example your output could be a simple OP_TRUE and spending this requires no unlocking script so it only adds the fixed size of the outpoint (transaction hash and index) + sequence. or the output could be a very complex P2SH script that has OP_IF, OP_CheckMultiSig,... and the signature script could become a lot bigger.
in most cases you are either using P2PKH or P2WPKH and since these require a single public key + a single signature the script size is small.
there is also the matter of the difference between weight/virtual size and size. we no longer measure transaction sizes in raw bytes but instead in virtual bytes. essentially this means the legacy inputs result in a bigger virtual size than SegWit inputs which is why you end up paying less fee when using P2WPKH compared to P2PKH with the same number of inputs and outputs.
there is some tools that you can play around with input/output and visualize your transaction size/fees. there is
https://coinb.in/#fees and a new one by
@bitmover https://bitcoindata.science/bitcoin-fee-size-calculator.html you can check out.