If a node tries to include a transaction with an UTXO which is already spent, marking it as a valid transaction, this node will get banned by other other nodes , as it is not following consensus rules.
This afaik only happens if the UTXO was already spent in a previous block, i.e. the "spending transaction" was already confirmed by a miner.
The real difficulty for nodes is to know to order double spends occurring in the same block. Let's imagine we have transaction X spending the same UTXO than transaction Y, and both are created only few milliseconds one after another. Let's say the person trying to double spend wants to damage a person or service in Australia who accepts 0-confirmation transactions, so they broadcast transaction X targeting specifically several nodes in Australia, hoping that the victim sees it first. At the same time they broadcast double spend Y to nodes close to many mining pools, e.g. in the US. So the Australian potential victim node "sees" transaction X and discards transaction Y, but most US miners do the opposite and include transaction Y instead, because they didn't see in their UTXO set that transaction X happened (as it is still unconfirmed and they received it after transaction Y).
As this is something that can happen all the time due to the latency between different networks (and more so if they're far away one from another, like in the example) I believe nodes won't ban another node for trying to broadcast an
unconfirmed double spend. They simply check if the transaction is consistent with their own UTXO set, and if not, they discard it.
Of course this "attack" is very unlikely to succeed, because it is unlikely that somebody really accepts a 0 confirmation transaction, and latency issues tend to improve with network speed. But it shows that the "chain state" and thus the "UTXO set" are always a local phenomenon, and all nodes manage it independently. This means in double spend cases nodes can't really see which transaction is the "valid" one, until a miner includes and thus confirms one of both transactions.
Most miners have very well-connected nodes that are geographically diverse. The reason for this is that they want to get started on a new block ASAP after a previous block has been found.
Similarly, many merchants have multiple nodes that are geographically diverse.
In both cases, the node owners try to avoid it being known the entity who is behind the node so to prevent attackers from being able to segment them from the rest of the network.
So spending two conflicting transactions at nearly the same time will not necessarily result in a particular transaction being confirmed. I would speculate that all else being equal, the transaction with the higher fee will be confirmed, and the other discarded in most cases.
Also, any entity accepting 0/unconfirmed transactions is likely to monitor the network for conflicting transactions, and not accept the transaction if a double spend attempt is detected.
