You can't imagine how not worried I am about someone scanning my ports and hacking me. I am sure there are a hundred other ways for someone to hack me if they want to. If you think the measures you are taking would stop a determined hacker, you are probably sorely mistaken.
Ehm, you are advertising those hundred other ways 24/7 by running the client 24/7. Not having open ports/services listening that aren't immediately needed is common security practice.
Also, just because you don't know there is no security hole in something doesn't mean there isn't one and there never will be. Blind faith and security by obscurity are bad practice. A port scanner may even store the info regarding your service in some database and return when an exploit is found.