What i mean is, isn't it possible to do like a transaction FROM an address and just plug in the right Trezor or Ledger and then it automatically finds the key for it, signs it (by approving on the device) and sends it?
That sounds exactly like using a "watching only" wallet + "offline signer" (air-gapped PC or hardware wallet) solution...
It would appear that your issue is that you need "coin control"/"spend from address" features + "Hardware wallet" support in the wallet software that you are using... for Ethereum. I know that what you're wanting would probably be possible with Electrum, but that is bitcoin only.
Electrum has really good coin control features and you can see a list of addresses (set to 25 by default, but possible to increase the gap limit if required)... and you can "spend from address". It also supports Trezor and Ledger.
Unfortunately, as far as I'm aware, no-one has created something similar for Ethereum... as most people just use MyEtherWallet or MyCrypto wallet... and most simply use one ETH address due to the way that network works.