So I'll fully admit, I'm NOT programmer enough for this project - I have a decent high-level understanding of a lot of what would be involved but I am nowhere near competent enough to actually write the code. Instead I'm just going to suggest an idea and see if someone who DOES have the skills will run with it. Perhaps if others like it enough we could even start a Bounty or something.
In light of the recent Mt. Gox issues I think it's time we built a distributed exchange. NameCoin has already proven that you can use a BTC-style blockchain to store non-coin data like DNS so why not provide a modified client that holds a buy/sell order database and matches buyers with sellers? The client could support a small variety of payment processors to begin (dwolla, etc) and expand with time. If the client interfaces directly with, for example, Dwolla's API then it should be able to transfer funds directly from my account to my seller's account, bypassing the middleman entirely.
I'm sure there is some obvious technical limitation that I'm missing or this would have been done already, right?
Why not have a smartphone app that matches up buyers and sellers of bitcoins IRL. Cash in hand for bitcoins? Perhaps this could work with your proposed system.