Definitely has to be the transaction time. That is really a pain and needs to be solved in my opinion as it leaves you with a long wait for confirmations.
Please give an example, because I don't see a problem.
I have spent bitcoin in 2 places: Overstock.com and Ocean Blue Sushi.
On Overstock.com, I checked out with Bitcoin, paid, and received my shipment in a few days. PayPal would not have been any faster.
At the restaurant, I asked to pay with bitcoin after the meal. The waitress brought a tablet device with a Bitcoin address and an amount (after I entered the tip amount). I paid with my mobile app (Mycelium), and after a few seconds she said: "Thank you for your payment."
In both the web and the brick-and-mortar cases, I didn't wait. I paid, and went on my way. What is the problem we are trying to solve here?