Each block has a pointer to the previous block, though, so how could this attack work? After you pay me in the block with timestamp X, you could forge another block with timestamp X-5 in which the money went somewhere else, but a traversal of the blockchain from the block with timestamp X would show that your new forged block was not in the chain, right?
What am I missing? I am speculating that it has something to do with chain forks and orphan blocks....
I think I get what you're saying. You're saying that because everyone already saw a certain block, it's impossible to forge a new one. However, it can easily be the case that because of network latency between peers, different node won't be in the same state at the same time. It is possible for an attacker to tell two different nodes about different versions of the same block and they won't hear of the other conflicting block until later. As DeathAndTaxes explained, this conflict of state is eventually resolved through proof-of-work.
Additionally, because nodes go online and offline in the network as they please, any system that depends on what was heard first without some way to resolve conflicts would totally collapse even if you had some way to reduce latency between all peers to zero.