It's possible mainly if you are looking for a decentralized exchange. The problem with centralized exchanges is that usually you can't use your own address and coins to make trades, since everything is happening in the back end and you don't want to give them your private-key. You need to deposit first and trade with a IOU. Meanwhile, DEX are making P2P transactions on your own to someone's else address, so you can load your private-key in the client and do the transactions.
I'm sure there are others, but this is one I tested and can confirm that works for Ledger and Trezor:
https://www.airswap.io/tradeIt's an ETH/tokens exchange. You connect your Ledger, put how much tokens you want to buy, confirm in the Ledger to send the ETH, and receive the tokens in the same address. Quick and secure.