Hardware wallet manufacturers don't want you to disassemble the devices. If you do that, you will lose the warranty. A physical inspection of the hardware components does make sense, though. People will have to weigh up the pros versus the cons and decide what's better for them personally. You either don't open the device and use it as it is. In that case the wallet is under warranty for a year or however long the warranty lasts. The other option is to take it apart to look inside and perhaps risk breaking something and you also lose your warranty by doing that.
There is one way to find out whether physical modifications have been made to the device.
Make the hardware wallet casing out of transparent plastic. Ledger offers several transparent casing options, but I don't think the plastic is clear enough to allow for a good view of the internal components.
HW manufacturers should further develop this transparent casing concept. It would help minimize the risks associated with modified devices without requiring any intrusion into the internal mechanism.
I know that Ledger shows on its website picture of how the inside boards and circuitry are supposed to look like in many of their models and generations. I don't know which other brands do that but it's definitely something they should do.
While I fully agree with this, malicious actors will also gain access to this information, which will allow them to better disguise their forgeries.
Also, the longer the supply chain, the greater the risk that someone may have manipulated a hardware wallet. If you buy it from the company directly, you are facing the lowest risk of physical manipulation. At the same time, your personal data is sitting in a centralized database that hackers and thieves will want to get their hands on. The alternative is to buy it from a reseller at a physical store to be safe from data leaks. However, you are then getting a product that has exchanged many more hands than if purchased directly from the manufacturer.
Every buyer will face this dilemma: either risk their personal data (exposing themselves to a threat) or risk the contents of their wallet (running the risk of buying a counterfeit).
This is really well thought out and maybe explains all those mysterious hacking cases where users claim they didn't do anything wrong but were hacked anyway. However, I think that it is not only Ledger that is exposed to this, it is probably not a problem to install such implants in other hardware wallets as well.
Ledger is probably the main target because they are the most known and biggest manufacturer of hardware wallets. It's unlikely that hackers would be interested in targeting a small company that sells much smaller quantities of their products.
I guess Trezor is their second most popular target. Shall we look forward to a "surprise" involving this brand?

Unfortunately, the only way to make sure something like this isn't installed in the device would be to open it up and inspect the components - but there are two problems with that - first, you lose your warranty, and second, most people aren't qualified to assess whether something has been added or not.
On the contrary, there is another (old) method. Can transfer a small amount of money to the "potentially counterfeit device" and watch to see if those funds move "on their own". This method is not 100% foolproof, however, as malicious actors might leave small amounts untouched (perhaps due to a minimum transaction threshold) and wait for a more substantial sum to appear.
It seems that we will all have to become engineers to protect ourselves from these attacks, or we will have to forget about hardware wallets as something that we have long considered the safest way to store private keys.
Securely managing cryptocurrencies requires a wide range of skills, including engineers ones.
One should not forget hardware wallets; however, it is essential to recognize that they don't offer a 100% guarantee of secure storage and to factor this into one's overall security strategy.