The timestamp on the block is not guaranteed to to be the correct time that the block was actually found. Miners can manipulate the block time quite a bit, so it doesn't reliably tell you when a block was found.
Propagation can also take a long time depending on how far away from the miners you are.
Note: I've already heard about FIBRE, but I'd like to achieve this without it. Also afaik FIBRE is about compact blocks and I'd also like to receive txns as fast as possible.
FIBRE is not really what you are looking for. You already are using compact blocks as that is in Bitcoin Core. Compact blocks is unrelated to transaction relay times.