One of the rules is that the fee for RBF transactions must increase. If you spam the transactions and don't increase the fee, they'll be rejected. So, at some point, you either run out of available coins to increase the fee, or one of the transactions gets confirmed which would immediately invalidate all the others and end the attack.
Additionally, if you send another transaction using RBF, the "original" transaction is then dropped from the mempool.
You can see the specs in BIP125:
One or more transactions currently in the mempool (original transactions) will be replaced by a new transaction (replacement transaction) that spends one or more of the same inputs if,
1. The original transactions signal replaceability explicitly or through inheritance as described in the above Summary section.
2. The replacement transaction may only include an unconfirmed input if that input was included in one of the original transactions. (An unconfirmed input spends an output from a currently-unconfirmed transaction.)
3. The replacement transaction pays an absolute fee of at least the sum paid by the original transactions.
4. The replacement transaction must also pay for its own bandwidth at or above the rate set by the node's minimum relay fee setting. For example, if the minimum relay fee is 1 satoshi/byte and the replacement transaction is 500 bytes total, then the replacement must pay a fee at least 500 satoshis higher than the sum of the originals.
5. The number of original transactions to be replaced and their descendant transactions which will be evicted from the mempool must not exceed a total of 100 transactions.
If you're not trying to RBF, and simply attempting to double-spend... then the transaction is generally just rejected outright, because you're trying to double-spend
you'll get "txn-conflict" type errors because a transaction spending the same UTXOs already exists.