Like the posters above said, as low as 0.2sat/vB should get your transaction confirmed. Are you sending from many small input because that's only when you need to pay more.
More number of inputs or outputs can make the transaction larger and make you pay more fee, but the required fee rate stays the same. It's not that you need to use a higher fee rate, if your transaction has more inputs aor outputs.
In other words, two transactions with the same fee rate has the same priority, regardless of the transaction virtual size. If 0.2 sat/vbyte is enough for getting confirmation, that's enough for any transaction, whether it has one input and 1 output or it has 100 inputs and 100 outputs.