If it is using the 'type-2' public derivation, e.g. as is the case for all keys in a current armory wallet (IIRC), and the attacker knows the extended public key (e.g. attacker has a watching wallet) then yes.
This is why in BIP32 the recommended top level uses the 'type-1' private derivation which doesn't have this surprising property (but also lacks the nifty ability for a untrusted party to generate addresses for the wallet).
That's why I think implementations should add an extra level of structure such that you create a different xpub for every entity from whom you receive funds.
I know, quadratic scaling, but it's worth it for the added safety.
