Bitcoin fees are really getting high, but it will cost you much when you’re using exchangers like Xapo, Coinbase, Binance and the rest of them, cause they will always more than the main Bitcoin wallet.
In some cases services are indeed asking more for withdrawals than a regular network fee would cost you, but that's something you have to accept because you make use of a centralized service. If you do so, you are basically subject to all their rules and laws, whether you like it or not. I however have to point out that Coinbase itself is pretty fair with the fees that they charge. Most exchanges use a fixed fee for withdrawals, while Coinbase adjusts its fees based on how congested the network is. High congestion means higher fee, and low congestion means lower fee.
The worst thing is that the transaction is not even fast like Litecoin.
You are talking about confirmation times I assume, because Bitcoin's transaction speed is pretty much instant. Bitcoin is the first ever crypto currency, so unlike Litecoin and other altcoins, Bitcoin didn't had an example of what could be done better or faster. Lightning Network will make sure things will play out in Bitcoin's favor. Also, if you guys start adopting Segwit, it would help the network even further, and at the same time help you enjoy lower transaction fees.