And, as there's just one miner, this one could reduce difficulty manually or even run a modified software.
It doesn't work that way. If you change the rules, the others don't have to follow you. If they encounter an unexpectedly low difficulty for a block, they wouldn't even accept it. They don't trust each other.
And yes, if we consider the use of SPV wallet (far most common and realistic), this attack would be impractical. But many bitcoiners consider everyone running full nodes promote decentralisation.
On the contrary, SPV clients are more vulnerable to this sort of attacks since they do not validate the whole block. Realistically, people are using Bitpay as a POS and you can't really do anything with this attack.