It is Public key Cryptography and only digital signatures, NO ENCRYPTION.
Isn't a digitally signed message the message encrypted with the private key?
No, encrypted messages are only created by using someone else's public key.
Your Bitcoin address is your public key.(speaking in pure practical terms)
Just a short example: You have a public key and a private key. Your public key can be shared on the internet so that people can send you Bitcoins. Your private key is kept your personal property(and you should keep it that way, otherwise people can spend your
BTC) and is used by you to digitally sign bitcoin transactions which you want to make.
Everyone else, actually the Bitcoin software, verify that your Bitcoin transaction is valid by checking your public key against the transaction that you signed with your private key.
I hope I didn't confuse you any, please ask more questions. It took me quite some time myself to get used to the general idea with private and public keys.