No, the IP/port the miners connect to can be different from the IP the pool broadcasts the finished block from.
The thought occurs to change mining -- to require an additional digital signature, so that knowledge of the secret key is required of the recipient address(es) that newly generated coins are to be output to, before a candidate block can be tested against the hash target, to see if it is a proper solution.
This makes it difficult to pool resources, since it ensures the 'solver' of the block has enough information to spend all the coins generated in the block that they solved.