When I store my coins on my Kraken account they hold the private key?
Yes they do. They only give you a public address so you can deposit on their wallets and use that balance on their website to sell/trade Bitcoins. Therefore, you don't really control the coins you have on their exchange.
I also have a wallet from MyEtherWallet, from which I do have a private key. The downside is that it seems that there are some coins that I can not hold in that wallet. Do you have multiple wallets for multiple coins? Seems so confusing..
Every coin has their own wallet. In your case, MyEtherWallet can only hold Ethereum coins and ERC-20 tokens.
If you want to store Bitcoins you will need to find and download a proper wallet for BTC. You can find the most recommended ones in this page:
https://bitcoin.org/en/choose-your-wallet. (I recommend you to try Electrum -
https://electrum.org)
If you don't want to download one wallet for each coin you have, there are a few wallets that let you manage multiple coins in a single software. If you are looking for this and have multiple coins to store, Exodus (
https://www.exodus.io) is a good choice.
They support those coins atm: Bitcoin, Augur, Dash, Dogecoin, Ethereum, Golem, Litecoin, and are planning to support more in the future.