The same goes for stock market. Why is it not decentralized yet, allowing people to exchange stocks without the interference of exchanges?
I think it might not happen. One reason is that an exchange is a good platform where there is high volumes of coins and security assured, another reason is that exchanges can be the eyes of a country's government to check how the trades and transactions are being done, so if there is anything suspicious they whisper the government to invesgate further.
Without interference of exchanges it would be a p2p market: it's risker to deal and it difficults the high volume transactions as the money will be fragmented. In addition it's hard for authorities to investigate frauds.