It all comes down to how you protect your private keys.
The private key gives you the ability to make bitcoin transactions, of what ever funds that is associated to your public key.
Anybody who has access to your private key has access to your bitcoin.
The only way you can be certain that you and only you have access to your bitcoin is to never expose the key to the internet.
I prefer to use paper-wallets, which is a type of offline wallet.
A paper-wallet is basically a piece of paper which has your private key written on.
If the paper-wallet is created completely offline and never exposed to the internet, then is not possible to steal your funds (unless someone break into your home and steal the physical piece of paper).
Note, you can always transfer funds to your paper-wallet, by sending funds to the associated public key. But you need the actual piece of paper to redeem the funds.
This exactly. If you have more than a "few" Bitcoins, I would just buy a Trezor. Mine's awesome, love it. No looking back at paper, other than my recovery seed list.
