To bring bitcoins to the physical world, we need a system that is largely available and cheap (exit the smartphones)
Smartphones are largely available and relatively cheap. As these are produced in massive quantities, unit prices are coming down. In parts of the developing world, sub-$100 androids are sold, for instance.
When you wanna buy something, you insert the card into the vendor reader. This reader is connected to a computer on wich a special software is running. The software, when a wallet.dat in a memory card pops in, run a bitcoind instance with this wallet, to send the agreed amount.
For security, the user has to enter is passphrase (much like he would enter a PIN with a credit card).
You are missing the next step. The vendor's card reader was hacked by Bob. Minutes after the customer leaves the store, all of the keys from the wallet.dat were used to spend the reaming balance of the wallet.
So that specific approach won't work.
There have been a number of approaches. Here's some discussion on the topic: