I took a look at it again and it doesn't mention how calendar time is handled. At least I couldn't find any information about it.
True. That particular question is not answered in the paper. And was answered very well in this thread.
The questions that are less well though out are:
Can a 55% attack on timestamps be done to mess up the blockchain?
Clearly not, since as can be seen in that paper, the time has no effect on the ordering of blocks in the blockchain. According to the paper, the blocks themselves in the blockchain are "the timestamp" that is used to order transactions (a block that occurs higher in the chain is "later" than a block that occurs lower in the chain regardless of any calendar time stored in the block).
But wouldn't the majority of nodes accept the false timestamps in a 55% attack? Hmm... And that would mean that they can even set timestamps backwards in time.
Since it had already been made clear that nodes will only accept a timestamp "if it is greater than the median timestamp of previous 11 blocks, and less than the network-adjusted time + 2 hours.", then it should be clear that a >50% attack can't change the timestamps significantly. The paper specifically states that nodes "reject invalid blocks by refusing to work on them." and that "Any needed rules and incentives can be enforced with this consensus mechanism." Furthermore, it states that "An attacker can only try to change one of his own transactions to take back money he recently spent".
Then how is double spending possible in a 51% attack?
This is well covered in section 11 of the paper.