Something important to note here: Handbrake is pretty terrible.
If you want to properly encode with Handbrake, then... actually you can't.
If you want to encode as best you can with Handbrake,
get a nightly build. Handbrake includes x264 as a library, not an executable, so you need the latest dev build to get the latest x264. And the latest x264 is always better; even the 3-4 month old build in Handbrake 0.9.5 is missing a bunch of bug fixes and some minor speed enhancements.
The 1.5 year old build in 0.9.4 is missing a
lot of very important features.