You're not wrong that a physical box can be a big blinking warning sign to thieves.
I have heard anecdotally from acquaintances in law enforcement about situations where a house has been burgled... the owners made insurance claims etc getting all new stuff like TVs/Laptops/Phones/Computers etc... and then put the boxes for all the new toys/gadgets/applicances out for recycling... basically, like a giant sign to the thieves "Hey, we got all new stuff!"
aaaaaaaaaand then they got burgled again a day or 2 later!
Having said that, it should be easy enough to dispose of the packaging for a hardware wallet without it being obvious... most come in relatively small packages... so destroying it shouldn't be that hard... a knife/scissors, or some fire would work well
and if you're paranoid that someone may be "dumpster diving" your trash, go for a walk around town and drop it in a random trash can or dispose of it in the trash at a McDonalds or something. Even if someone finds it, it won't be able to be tracked back to your address.