I honestly think a lot of the 'average joe' preconceptions are around 10yrs outdated.
High-Tech stuff is -everywhere-. On TV, in your home, in movies, out in public. People may not be familiar with it all, but they aren't cavemen.
Digital money is already here. People know how to use online-banking, credit cards, etc. The -only- thing that would make it difficult for the 'average joe' to pickup Bitcoin is the wallet security. I *think* I read that in the bitcoin code, there's the option to enable multi-party signatures to access a wallet. That with a (credible) mainstream online wallet, and that problem is solved.
Online money has been made super easy to use and it's safe enough, so lots of people use it.
Bitcoins are not that easy to get or use so unless there is some reason beyond the novelty, average people have no reason to be bothered.
You're right, it's not about tech savviness, it's about whether it is easy and safe. When Bitcoins are both easy to use and safe to use, then the "average joe" will start to use them.