If that is the case, hypothetically taking the blocks as attacks, it could cause one or two people to complain about delays and generally cause a bitcoin price fluctuation that allows the attacker to aggressively enter the market beyond just mining blanks.
"empty" blocks (like all other blocks) don't delay non-empty blocks from being solved
to be precise, after the difficulty readjusts, they would delay waiting for a non-empty block by
(10 min) * (their share of the network hashrate during the previous 2016 blocks)
which is negligible