There's not even any need to pull the disk. Without any encryption it is trivial to boot into the root account, change any password, collect any files, install any software, and put the passwords back when you're done if you want.
With home directory encryption your personal files would be safe, nobody is going to get access to those unless they have your password. Of course, if they have repeated physical access while it's in you possession they could boot into the root account and install a key-logger.
You don't have to browse shady things to be a potential target. For example, compromised ad servers can push attacks across many popular respectable sites.
There are no guarantees, just levels of confidence.
you could also do some of the coldboot voodoo, reading from coldbooted ram attacks, to get the password.