I suggest sticking with handbrake. First take a 30 second video choose one setting, then change other setting and compare them. So it is a trail and error method. When you feel like you got a good balance between the size and quality keep that settings and encode whole video or series. After doing that several times you get a hang of it and you will know what to set to get the video you desire. Note that handbrake doesn't have filters, so any errors in videos can't be corrected.
After getting little experience you can go to staxrip which has filters like sharpening, denoise etc. To reduce the errors.
First you need to do a lot of reading though, I mean a lot.