This actually isn't desirable. In the current system miners have no economic incentive to lie about their timestamps (no marginal gain from doing so), except for the window boundary block, whos permissible times are constrained by the surrounding blocks and the actual time, allowing at most a small one time difficulty shift (assuming the warping bug is fixed). Including other blocks in the time computation would create an incentive to lie constantly for every miner, and for the same final response time a much larger allowable skew.

That's no different than the incentive in the current system, however. The existing constraint on block header timestamps vs. network time caps the amount of michief that can be done in either case. But with the harmonic mean

*every* miner would have to lie to achieve the same effect as just the miner of the first/last block being dishonest in the arithmetic mean scenario.

This absolutely cannot work. The harmonic expectation of the exponential distribution is 0, so there is no way to retarget using it.

Even if you use a geometric mean or something else that does converge, it will have very high variance causing retargets to be all over the place. Not only that, but an attacker could release blocks with reported timestamp a second after the previous ones, causing a massive drop in the geometric mean and a spike in the difficulty.

Are you talking about continuous difficulty adjustment? Otherwise I'm really not sure where you're coming from. The question in the OP is about a drop-in replacement of the method for determining the average hashrate in bitcoin's once-every-2016th-block difficulty adjustment. I assure you it does work as I have live code doing it right now, and it retargets just fine; it even results in a more accurate adjustment. The harmonic mean, not the arithmetic mean, yields the actual average hashrate for the last N blocks, although the difference between the two is typically very small.

Also, messing with the timestamp of the last block may have some short-term effect, but not so much on the long-term because of the rules for validity of timestamps.

The question is more theoretical than anything. Such mischief would result in at most an increase or decrease in the difficulty by a few percent, and would likely be corrected in the next round anyway. Really I'm asking the question out of a desire to understand: I'm wondering if there is a reason Satoshi decided to use the arithmetic mean (such as preventing an attack or removing incentive for bad behavior), or if it was just a mistake (a very common mistake, to be fair).