Transaction time has nothing to do with confirmation time.
CC transaction is instant but takes 60 days to confirm.
It does matter when you have a retail transaction and you don't know who your customer is, for example the food markets in developing countries where any one can walk in and buy some fresh meat or vegetables. The vendor is moving fast. And most people don't have identification any way.
Also it would matter in retail transactions in the developed countries where fake identification would even be possible in some cases.
And do we really want to move to a world where we carry our identification on an RFID chip in our body in order to make it efficient!
True the merchant could accept the cost of fraud as a cost of doing business, as is the case with online merchants which accept credit cards. However, note that many retail merchants operate on razor thin margins. WalMart prefers debit cards I think.
Thus it is erroneous to assert that 1-confirmation delay isn't relevant in all cases.
And off-chain transactions do indeed bring us back to the issues of centralization required to deal with (abuse of) trust, which is of course a slippery slope just as how putting our gold on deposit in the 1800s and trading in certificates lead us to bank failures (due to fractional reserves lending of certificates, i.e. cheating) and then finally a central bank to prevent corrections (i.e. prevent debt defaults) and keep the debt increasing non-stop for the past 80 years or so without correction (thus leading to a horrific global implosion coming before 2020).