For that matter, when I buy a hardware wallet, how can I trust it to have legit private key? Is there way to verify the legitimacy of the private key?
What do you mean by "legit" private key? That it is only known to you and no one else? Then it's a matter of trusting the software/hardware that generated the key (a key is just a big random number). Or, if it's open source, then you can verify it yourself, though you need to be proficient in programming to do that. But a lot of other people have probably done it, so you can just check if everything is alright with the latest version of a wallet.
What is more important is to check that the wallet (either software or hardware) is genuine and wasn't tampered with. With software you need to check the digital signatures before installing it, with hardware wallet you should follow the vendor instructions to verify integrity.