But I don't understand why. More specifically, why would miners not replace three transactions at 10 sat/vB (for example) by a single one at 11 sat/vB ? Because of course it's less fees overall, but also less space used, so the important metric should be the fee/space_used ratio (that is, the sat/vB rate). What am I missing?
Great question. There are two main reasons why a replacement transaction must pay higher fees than all the unconfirmed transactions it is replacing.
The first reason you almost figured out yourself in the sentence I quoted above. Taking your example - let's say I have a chain of three unconfirmed transactions paying 10 sats/vbyte each. I want to replace the first one with a new transaction paying 11 sats/vbyte, which would evict the other two, freeing up space for the miner to include other transactions. But maybe the next highest fee paying transactions in the mempool which the miner can include only pay 3 sats/vbyte. It then makes more sense for the miner to ignore my replacement and mine my three original transactions at 10 sats/vbyte each, than it does for them to accept my replacement and mine one transaction at 11 sats/vbyte and two transactions at 3 sats/vbyte. My replacement must therefore pay more than all the transactions it is evicting to ensure it is always economical for miners to accept it.
The second reason is DoS protection. Let's say I broadcast a transaction at 1 sat/vbyte when the mempool is quite full and I know it isn't going to be confirmed any time soon. I then broadcast 20 more transactions building on this transaction, with a combined size of tens of thousands of bytes, all paying 1 sat/vbyte. Every node in the network has to receive, validate, and then broadcast my huge transactions. I can then immediately cancel all of those transactions by replacing my original transaction at a fee of 1.01 sats/vbyte. I can then broadcast another 20 more new child transactions with a huge size building on this replacement, which again, every node has to receive, validate, and broadcast. I can then cancel them all by replacing my original transaction again with a new one paying 1.02 sats/vbyte. I can repeat this thousands of times for almost zero cost and completely spam out the network. But by making me pay a fee higher than all the transactions I am evicting, this kind of attack rapidly becomes prohibitively expensive.