For one thing, your site isn't even SSL capable, so even if you are entirely honest, an attacker could alter the contents of the traffic between you and me, and I could end up downloading a different file than what you are hosting.
Even with SSL the thing you state is possible, it's not SSL per se
that forbid a man in the middle attack, you also need authentication.
He could get a valid SSL certificate signed by some central issuer, but even in that case the government can ask (and obtain) the issuer to issue for him a valid certificate with the same credentials so he can execute a man in the middle attack with a completely unaware user.
SSL prevents eavesdrop to casual sniffers but there is really no point in doing so with a public available file :-)
So, as you correctly stated, the only way to prevent this is to digitally sign the files so they can be checked using the public key that one must get via other channels.
EDIT: foxpup, you made it quicker than me ;-)